[jira] [Work logged] (HIVE-25893) NPE when reading Parquet data because ColumnVector isNull[] is not updated
[
https://issues.apache.org/jira/browse/HIVE-25893?focusedWorklogId=715457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715457
]
ASF GitHub Bot logged work on HIVE-25893:
-
Author: ASF GitHub Bot
Created on: 26/Jan/22 06:07
Start Date: 26/Jan/22 06:07
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #2970:
URL: https://github.com/apache/hive/pull/2970#discussion_r792335168
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java
##
@@ -500,22 +501,37 @@ private boolean
compareDecimalColumnVector(DecimalColumnVector cv1, DecimalColum
private boolean compareBytesColumnVector(BytesColumnVector cv1,
BytesColumnVector cv2) {
int length1 = cv1.vector.length;
int length2 = cv2.vector.length;
-if (length1 == length2) {
- for (int i = 0; i < length1; i++) {
-int innerLen1 = cv1.vector[i].length;
-int innerLen2 = cv2.vector[i].length;
-if (innerLen1 == innerLen2) {
- for (int j = 0; j < innerLen1; j++) {
-if (cv1.vector[i][j] != cv2.vector[i][j]) {
- return false;
-}
- }
-} else {
+if (length1 != length2) {
+ return false;
+}
+
+for (int i = 0; i < length1; i++) {
+ // check for different nulls
+ if (columnVectorsDifferNullForSameIndex(cv1, cv2, i)) {
+return false;
+ }
+
+ // if they are both null, continue
+ // else if one of them is null, return false
+ if (cv1.isNull[i] && cv2.isNull[i]) {
+continue;
+ } else if (cv1.isNull[i] || cv2.isNull[i]) {
Review comment:
This check seems to be unnecessary because it is already handled by
```
if (columnVectorsDifferNullForSameIndex(cv1, cv2, i)) {
return false;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 715457)
Time Spent: 0.5h (was: 20m)
> NPE when reading Parquet data because ColumnVector isNull[] is not updated
> --
>
> Key: HIVE-25893
> URL: https://issues.apache.org/jira/browse/HIVE-25893
> Project: Hive
> Issue Type: Bug
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In
> [VectorizedListColumnReader.java|https://github.com/apache/hive/blob/595f3bc9d612f02581bd3377ee0107efd6553ae6/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java]
> {{isNull[]}} is used in the comparison methods ( eg.
> [columnVectorsDifferNullForSameIndex
> |https://github.com/apache/hive/blob/595f3bc9d612f02581bd3377ee0107efd6553ae6/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java#L524]
> ), however, {{isNull}} is always {{false}} as it is never updated in
> [getChildData|https://github.com/apache/hive/blob/595f3bc9d612f02581bd3377ee0107efd6553ae6/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java#L401].
> This could result in NullPointerException like,
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.compareBytesColumnVector(VectorizedListColumnReader.java:506)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.compareColumnVector(VectorizedListColumnReader.java:432)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.setIsRepeating(VectorizedListColumnReader.java:367)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.convertValueListToListColumnVector(VectorizedListColumnReader.java:360)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.readBatch(VectorizedListColumnReader.java:83)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedMapColumnReader.readBatch(VectorizedMapColumnReader.java:57)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:438)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:377)
> at
>
[jira] [Work logged] (HIVE-25809) Implement URI Mapping for KuduStorageHandler in Hive
[ https://issues.apache.org/jira/browse/HIVE-25809?focusedWorklogId=715418=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715418 ] ASF GitHub Bot logged work on HIVE-25809: - Author: ASF GitHub Bot Created on: 26/Jan/22 01:53 Start Date: 26/Jan/22 01:53 Worklog Time Spent: 10m Work Description: nrg4878 commented on pull request #2877: URL: https://github.com/apache/hive/pull/2877#issuecomment-1021782259 Fix has been merge to master. Please close the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 715418) Time Spent: 20m (was: 10m) > Implement URI Mapping for KuduStorageHandler in Hive > - > > Key: HIVE-25809 > URL: https://issues.apache.org/jira/browse/HIVE-25809 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, there is no storage URI mapping for KuduStorageHandler based on > the feature HIVE-24705. The API getURIForAuth() needs to be implemented in > KuduStorageHandler. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25809) Implement URI Mapping for KuduStorageHandler in Hive
[ https://issues.apache.org/jira/browse/HIVE-25809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-25809. -- Target Version/s: 4.0.0 Resolution: Fixed Fix has been merged to master. Please close the PR. Thank you for the fix [~hemanth619] > Implement URI Mapping for KuduStorageHandler in Hive > - > > Key: HIVE-25809 > URL: https://issues.apache.org/jira/browse/HIVE-25809 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, there is no storage URI mapping for KuduStorageHandler based on > the feature HIVE-24705. The API getURIForAuth() needs to be implemented in > KuduStorageHandler. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25143) Improve ERROR Logging in QL Package
[
https://issues.apache.org/jira/browse/HIVE-25143?focusedWorklogId=715369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715369
]
ASF GitHub Bot logged work on HIVE-25143:
-
Author: ASF GitHub Bot
Created on: 26/Jan/22 00:13
Start Date: 26/Jan/22 00:13
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #2301:
URL: https://github.com/apache/hive/pull/2301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 715369)
Time Spent: 50m (was: 40m)
> Improve ERROR Logging in QL Package
> ---
>
> Key: HIVE-25143
> URL: https://issues.apache.org/jira/browse/HIVE-25143
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I went through and reviewed all of the ERROR logging in the HS2 {{ql}} module
> and I removed (most of) the following bad habits:
>
> * Log-and-Throw (log or throw, not both)
> * Pass in the Exception to the logging framework instead of logging its
> toString() : LOG.error("alter table update columns: {}", e);
> * Add additional context instead of copying the message from the wrapped
> Exception : throw new SemanticException(e.getMessage(), e);
> * The wrapped exception is being lost in some case, though the message
> survives : throw new HiveException(e.getMessage());
> * Remove new-lines from Exception messages, this is annoying as log messages
> should all be on a single line for GREP
> * Not logging the Exception stack trace : LOG.error("Error in close loader:
> " + ie);
> * Logging information but not passing it into an Exception for bubbling up:
> LOG.error("Failed to return session: {} to pool", session, e); throw e;
> * Other miscellaneous improvements
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-20723) Allow per table specification of compaction yarn queue
[
https://issues.apache.org/jira/browse/HIVE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482124#comment-17482124
]
Stamatis Zampetakis commented on HIVE-20723:
It seems that it was possible to select a queue per table without implementing
the change here by using {{compactor.mapred.job.queue.name}} inside table
properties.
[~ekoifman] [~saurabhseth] is there any particular reason of why you chose to
do it via the {{compactor.hive.compactor.job.queue}} property? The problem is
that with the change in this JIRA the old property gets overwritten and doesn't
have the desired effect so it kinda breaks backward compatibility.
> Allow per table specification of compaction yarn queue
> --
>
> Key: HIVE-20723
> URL: https://issues.apache.org/jira/browse/HIVE-20723
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20723.patch
>
>
> Currently compactions of full CRUD transactional tables are Map-Reduce jobs
> submitted to a yarn queue defined by hive.compactor.job.queue property.
> If would be useful to be able to override this on per table basis by putting
> it into table properties so that compactions for different tables can use
> different queues.
>
> There is already ability to override other compaction related configs via
> table props, though this will need additional handling to set the queue name
> {{CompactorMr.createBaseJobConf}}
> [https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-TableProperties]
>
> See {{CopactorMR.COMPACTOR_PREFIX}} and
> {{Initiator.COMPACTORTHRESHOLD_PREFIX}}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25575) Add support for JWT authentication
[ https://issues.apache.org/jira/browse/HIVE-25575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-25575: Assignee: Yu-Wen Lai (was: Shubham Chaurasia) > Add support for JWT authentication > -- > > Key: HIVE-25575 > URL: https://issues.apache.org/jira/browse/HIVE-25575 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, JDBC >Affects Versions: 4.0.0 >Reporter: Shubham Chaurasia >Assignee: Yu-Wen Lai >Priority: Major > > It would be good to support JWT auth mechanism in hive. In order to implement > it, we would need the following - > On HS2 side - > 1. Accept JWT in Authorization: Bearer header. > 2. Fetch JWKS from a public endpoint to verify JWT signature, to start with > we can fetch on HS2 start up. > 3. Verify JWT Signature. > On JDBC Client side - > 1. Hive jdbc client should be able to accept jwt in JDBC url. (will add more > details) > 2. Client should also be able to pick up JWT from an env var if it's defined. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-23648) Fix FindBug issues in hive-serde
[ https://issues.apache.org/jira/browse/HIVE-23648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Bathori updated HIVE-23648: Status: Patch Available (was: Open) > Fix FindBug issues in hive-serde > > > Key: HIVE-23648 > URL: https://issues.apache.org/jira/browse/HIVE-23648 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Mark Bathori >Priority: Major > Labels: pull-request-available > Attachments: spotbugsXml.xml > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-23648) Fix FindBug issues in hive-serde
[ https://issues.apache.org/jira/browse/HIVE-23648?focusedWorklogId=715198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715198 ] ASF GitHub Bot logged work on HIVE-23648: - Author: ASF GitHub Bot Created on: 25/Jan/22 18:29 Start Date: 25/Jan/22 18:29 Worklog Time Spent: 10m Work Description: mbathori-cloudera commented on pull request #2969: URL: https://github.com/apache/hive/pull/2969#issuecomment-1021486016 The TestMiniLlapLocalCliDriver [avro_schema_evolution_native] test failure seems to be an intermittent issue. It is passing locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 715198) Time Spent: 20m (was: 10m) > Fix FindBug issues in hive-serde > > > Key: HIVE-23648 > URL: https://issues.apache.org/jira/browse/HIVE-23648 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: spotbugsXml.xml > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-23648) Fix FindBug issues in hive-serde
[ https://issues.apache.org/jira/browse/HIVE-23648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Bathori reassigned HIVE-23648: --- Assignee: Mark Bathori > Fix FindBug issues in hive-serde > > > Key: HIVE-23648 > URL: https://issues.apache.org/jira/browse/HIVE-23648 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Mark Bathori >Priority: Major > Labels: pull-request-available > Attachments: spotbugsXml.xml > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[ https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=715110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715110 ] ASF GitHub Bot logged work on HIVE-25892: - Author: ASF GitHub Bot Created on: 25/Jan/22 17:14 Start Date: 25/Jan/22 17:14 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2967: URL: https://github.com/apache/hive/pull/2967#issuecomment-1020861747 Could we provide a way to clear the context? In Iceberg we create a HMS instance for our tests, and because of the threadlocals here and in the TxnHandler we leak information between the tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 715110) Time Spent: 3h 20m (was: 3h 10m) > Group HMSHandler's thread locals into a single context > -- > > Key: HIVE-25892 > URL: https://issues.apache.org/jira/browse/HIVE-25892 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > There are more than six ThreadLocal variables in HMSHandler, we can group > them together into a single context to improve the management of variables > and the code readability. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=715108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715108
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 17:13
Start Date: 25/Jan/22 17:13
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791589935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -169,65 +169,29 @@
private Warehouse wh; // hdfs warehouse
private static Striped tablelocks;
- private static final ThreadLocal threadLocalMS = new
ThreadLocal();
- private static final ThreadLocal threadLocalTxn = new
ThreadLocal();
-
- private static final ThreadLocal> timerContexts =
- new ThreadLocal>() {
-@Override
-protected Map
initialValue() {
- return new HashMap<>();
-}
- };
-
public static RawStore getRawStore() {
-return threadLocalMS.get();
+return HMSHandlerContext.getRawStore().orElse(null);
}
static void cleanupRawStore() {
Review comment:
It's been used only in two places(HMSHandler) when client
disconnects，can we rename it to `cleanupContext` with public modifier? Thank
you for the suggestions!
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -169,65 +169,29 @@
private Warehouse wh; // hdfs warehouse
private static Striped tablelocks;
- private static final ThreadLocal threadLocalMS = new
ThreadLocal();
- private static final ThreadLocal threadLocalTxn = new
ThreadLocal();
-
- private static final ThreadLocal> timerContexts =
- new ThreadLocal>() {
-@Override
-protected Map
initialValue() {
- return new HashMap<>();
-}
- };
-
public static RawStore getRawStore() {
-return threadLocalMS.get();
+return HMSHandlerContext.getRawStore().orElse(null);
}
static void cleanupRawStore() {
Review comment:
It's been used only in two places(HMSHandler) when client
disconnects，can we rename it to `cleanupHandlerContext` with public modifier?
Thank you for the suggestions!
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandlerContext.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+
+/**
+ * When one hms client connects in, we create a handler context for it.
+ * We store session information here.
+ */
+public final class HMSHandlerContext {
+
+ private static final ThreadLocal context = new
ThreadLocal<>();
+
+ private static final AtomicInteger nextSerialNum = new AtomicInteger();
+
+ private RawStore rawStore;
+
+ private TxnStore txnStore;
+
+ // Thread local HMSHandler used during shutdown to notify meta listeners
+ private HMSHandler hmsHandler;
+
+ // Thread local configuration is needed as many threads could make changes
+ // to the conf using the connection hook
+ private Configuration configuration;
+
+ // Thread local Map to keep track of modified meta conf keys
+ private Map modifiedConfig = new HashMap<>();
+
+ private Integer threadId = nextSerialNum.incrementAndGet();
+ // This will only be set if the metastore is being accessed from a metastore
Thrift server,
+ // not if it is from the CLI. Also, only if the TTransport being used to
connect is an
+ // instance of TSocket. This is also not set when kerberos is used.
+ private String ipAddress;
+
+ private Map timerContexts = new
HashMap<>();
+
+ private HMSHandlerContext() {
+
+ }
+
+ public
[jira] [Work logged] (HIVE-25879) MetaStoreDirectSql test query should not query the DBS table
[
https://issues.apache.org/jira/browse/HIVE-25879?focusedWorklogId=715051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715051
]
ASF GitHub Bot logged work on HIVE-25879:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 17:09
Start Date: 25/Jan/22 17:09
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #2964:
URL: https://github.com/apache/hive/pull/2964#issuecomment-1021002371
@mszurap: Could you please separate the 2 changes?
Could you please make sure that the tests are green?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 715051)
Time Spent: 0.5h (was: 20m)
> MetaStoreDirectSql test query should not query the DBS table
>
>
> Key: HIVE-25879
> URL: https://issues.apache.org/jira/browse/HIVE-25879
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Szurap
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The runTestQuery() in the
> org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java is using a test query
> {code:java}
> select "DB_ID" from "DBS"{code}
> to determine whether the direct SQL can be used.
> With larger deployments with many (10k+) Hive databases it would be more
> efficienct to query a small table instead, for example the "VERSION" table
> should always have a single row only.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25897) Move delta metric collection into AcidMetricsService
[ https://issues.apache.org/jira/browse/HIVE-25897?focusedWorklogId=715038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-715038 ] ASF GitHub Bot logged work on HIVE-25897: - Author: ASF GitHub Bot Created on: 25/Jan/22 17:08 Start Date: 25/Jan/22 17:08 Worklog Time Spent: 10m Work Description: lcspinter opened a new pull request #2973: URL: https://github.com/apache/hive/pull/2973 ### What changes were proposed in this pull request? Refactor AcidMetricsService to collect delta metrics as well. ### Why are the changes needed? DeltaFilesMetricReporter and AcidMetricsService are two different threads collecting ACID related metrics. It makes sense to merge those threads since they share the same goal. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test, unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 715038) Time Spent: 20m (was: 10m) > Move delta metric collection into AcidMetricsService > > > Key: HIVE-25897 > URL: https://issues.apache.org/jira/browse/HIVE-25897 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > DeltaFilesMetricReporter and AcidMetricsService are two different threads > collecting ACID related metrics. It makes sense to merge those threads since > they share the same goal. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714961
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 17:01
Start Date: 25/Jan/22 17:01
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791581995
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -169,65 +169,29 @@
private Warehouse wh; // hdfs warehouse
private static Striped tablelocks;
- private static final ThreadLocal threadLocalMS = new
ThreadLocal();
- private static final ThreadLocal threadLocalTxn = new
ThreadLocal();
-
- private static final ThreadLocal> timerContexts =
- new ThreadLocal>() {
-@Override
-protected Map
initialValue() {
- return new HashMap<>();
-}
- };
-
public static RawStore getRawStore() {
-return threadLocalMS.get();
+return HMSHandlerContext.getRawStore().orElse(null);
}
static void cleanupRawStore() {
Review comment:
This is not a `cleanupRawStore` for a good while...
How widely used is this? Could we rename this?
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandlerContext.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+
+/**
+ * When one hms client connects in, we create a handler context for it.
+ * We store session information here.
+ */
+public final class HMSHandlerContext {
+
+ private static final ThreadLocal context = new
ThreadLocal<>();
+
+ private static final AtomicInteger nextSerialNum = new AtomicInteger();
+
+ private RawStore rawStore;
+
+ private TxnStore txnStore;
+
+ // Thread local HMSHandler used during shutdown to notify meta listeners
+ private HMSHandler hmsHandler;
+
+ // Thread local configuration is needed as many threads could make changes
+ // to the conf using the connection hook
+ private Configuration configuration;
+
+ // Thread local Map to keep track of modified meta conf keys
+ private Map modifiedConfig = new HashMap<>();
+
+ private Integer threadId = nextSerialNum.incrementAndGet();
+ // This will only be set if the metastore is being accessed from a metastore
Thrift server,
+ // not if it is from the CLI. Also, only if the TTransport being used to
connect is an
+ // instance of TSocket. This is also not set when kerberos is used.
+ private String ipAddress;
+
+ private Map timerContexts = new
HashMap<>();
+
+ private HMSHandlerContext() {
+
+ }
+
+ public static Optional getRawStore() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalRawStore() : Optional.empty();
+ }
+
+ public static Optional getHMSHandler() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalHmsHandler() : Optional.empty();
+ }
+
+ public static Optional getIpAddress() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getRemoteIpAddress() : Optional.empty();
+ }
+
+ public static Optional getConfiguration() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalConfiguration() : Optional.empty();
+ }
+
+ public static TxnStore getTxnStore(Configuration conf) {
+return getContext().getLocalTxnStore().orElseGet(() -> {
+ TxnStore txnStore = TxnUtils.getTxnStore(conf);
+ setTxnStore(txnStore);
+ return txnStore;
+});
+ }
+
+ public static Map getModifiedConfig() {
+return getContext().modifiedConfig;
+ }
+
+ public static Integer
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[ https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714918 ] ASF GitHub Bot logged work on HIVE-25892: - Author: ASF GitHub Bot Created on: 25/Jan/22 16:57 Start Date: 25/Jan/22 16:57 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2967: URL: https://github.com/apache/hive/pull/2967#issuecomment-1020924354 > Could we provide a way to clear the context? In Iceberg we create a HMS instance for our tests, and because of the threadlocals here and in the TxnHandler we leak information between the tests. Add a [CleanupHook](https://github.com/apache/hive/blob/c8c6152b79810276ac070ec057b0643b9ce3b03f/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandlerContext.java#L170-L172) here to clear the locals by calling HMSHandlerContext#clear(CleanupHook cleanupHook), not sure it enough for the iceberg case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 714918) Time Spent: 2h 50m (was: 2h 40m) > Group HMSHandler's thread locals into a single context > -- > > Key: HIVE-25892 > URL: https://issues.apache.org/jira/browse/HIVE-25892 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > There are more than six ThreadLocal variables in HMSHandler, we can group > them together into a single context to improve the management of variables > and the code readability. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25706) ShuffleHandler: Make sure of properly releasing netty reference counted objects
[
https://issues.apache.org/jira/browse/HIVE-25706?focusedWorklogId=714905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714905
]
ASF GitHub Bot logged work on HIVE-25706:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 16:56
Start Date: 25/Jan/22 16:56
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #2794:
URL: https://github.com/apache/hive/pull/2794
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714905)
Time Spent: 1h (was: 50m)
> ShuffleHandler: Make sure of properly releasing netty reference counted
> objects
> ---
>
> Key: HIVE-25706
> URL: https://issues.apache.org/jira/browse/HIVE-25706
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> https://netty.io/wiki/reference-counted-objects.html
> ShuffleHandler log contains issues below when we're on unhappy codepath
> (sendError). Very easily reproducible e.g. with calling ShuffleHandler's http
> port without any parameters. Doesn't cause a user-facing issue, because at
> this point ShuffleHandler already responded to the client, but the exception
> is annoying and implies invalid .release() calls in netty4.
> {code}
> io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
> at
> io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.handler.codec.http.DefaultFullHttpResponse.release(DefaultFullHttpResponse.java:116)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.sendError(ShuffleHandler.java:1056)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.handleRequest(ShuffleHandler.java:827)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.channelRead(ShuffleHandler.java:728)
> ~[classes/:?]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25883) Enhance Compaction Cleaner to skip when there is nothing to do
[ https://issues.apache.org/jira/browse/HIVE-25883?focusedWorklogId=714804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714804 ] ASF GitHub Bot logged work on HIVE-25883: - Author: ASF GitHub Bot Created on: 25/Jan/22 16:49 Start Date: 25/Jan/22 16:49 Worklog Time Spent: 10m Work Description: kgyrtkirk opened a new pull request #2971: URL: https://github.com/apache/hive/pull/2971 (cherry picked from commit 4c742cd5b0c523e1c102faa109b66bd92aac2ece) ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 714804) Time Spent: 1h (was: 50m) > Enhance Compaction Cleaner to skip when there is nothing to do > -- > > Key: HIVE-25883 > URL: https://issues.apache.org/jira/browse/HIVE-25883 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > the cleaner works the following way: > * it identifies obsolete directories (delta dirs ; which doesn't have open > txns) > * removes them and done > if there are no obsolete directoris that is attributed to that there might be > open txns so the request should be retried later. > however if for some reason the directory was already cleaned - similarily it > has no obsolete directories; and thus the request is retried for forever -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25706) ShuffleHandler: Make sure of properly releasing netty reference counted objects
[
https://issues.apache.org/jira/browse/HIVE-25706?focusedWorklogId=714768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714768
]
ASF GitHub Bot logged work on HIVE-25706:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 16:46
Start Date: 25/Jan/22 16:46
Worklog Time Spent: 10m
Work Description: abstractdog opened a new pull request #2794:
URL: https://github.com/apache/hive/pull/2794
### What changes were proposed in this pull request?
Release the ByteBuf instance at the last usage point instead of releasing
the http response.
### Why are the changes needed?
It threw exceptions silently in ShuffleHandler logs, easily reproducible in
TestShuffleHandler.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Rerun TestShuffleHandler, IllegalReferenceCountException disappeared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714768)
Time Spent: 50m (was: 40m)
> ShuffleHandler: Make sure of properly releasing netty reference counted
> objects
> ---
>
> Key: HIVE-25706
> URL: https://issues.apache.org/jira/browse/HIVE-25706
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> https://netty.io/wiki/reference-counted-objects.html
> ShuffleHandler log contains issues below when we're on unhappy codepath
> (sendError). Very easily reproducible e.g. with calling ShuffleHandler's http
> port without any parameters. Doesn't cause a user-facing issue, because at
> this point ShuffleHandler already responded to the client, but the exception
> is annoying and implies invalid .release() calls in netty4.
> {code}
> io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
> at
> io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.handler.codec.http.DefaultFullHttpResponse.release(DefaultFullHttpResponse.java:116)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.sendError(ShuffleHandler.java:1056)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.handleRequest(ShuffleHandler.java:827)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.channelRead(ShuffleHandler.java:728)
> ~[classes/:?]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25405) Implement Connector Provider for Amazon Redshift
[
https://issues.apache.org/jira/browse/HIVE-25405?focusedWorklogId=714767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714767
]
ASF GitHub Bot logged work on HIVE-25405:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 16:46
Start Date: 25/Jan/22 16:46
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #2546:
URL: https://github.com/apache/hive/pull/2546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714767)
Time Spent: 1h 20m (was: 1h 10m)
> Implement Connector Provider for Amazon Redshift
>
>
> Key: HIVE-25405
> URL: https://issues.apache.org/jira/browse/HIVE-25405
> Project: Hive
> Issue Type: Sub-task
>Reporter: Narayanan Venkateswaran
>Assignee: Narayanan Venkateswaran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> h2. +DB Level Federation to Redshift+
> The work in this Jira helps setup federation to redshift databases at the DB
> level, in the following way.
> {noformat}
> 0: jdbc:hive2://> describe connector redshift_test_7;
> 21/07/30 20:53:53 [343005b1-06f3-4891-81d1-0c6ad8612260 main]: WARN
> lazy.LazyStruct: Extra bytes detected at the end of the row! Ignoring similar
> problems.
> +--+---++-+-+--+
> | name | type|url
> | owner_name | owner_type | comment |
> +--+---++-+-+--+
> | redshift_test_7 | redshift | jdbc:redshift://
> | narayanan | USER| test redshift connector |
> +--+---++-+-+--+
> 1 row selected (0.054 seconds) {noformat}
> {noformat}
> 0: jdbc:hive2://> describe database db_redshift_7;
> 21/07/30 20:53:38 [343005b1-06f3-4891-81d1-0c6ad8612260 main]: WARN
> lazy.LazyStruct: Extra bytes detected at the end of the row! Ignoring similar
> problems.
> ++--+---+--+-+-+--++
> |db_name | comment | location | managedlocation | owner_name |
> owner_type | connector_name | remote_dbname |
> ++--+---+--+-+-+--++
> | db_redshift_7 | | | | narayanan |
> USER| redshift_test_7 | dev|
> ++--+---+--+-+-+--++
> 1 row selected (0.066 seconds) {noformat}
> {noformat}
> 0: jdbc:hive2://> use db_redshift_7;
> No rows affected (1.391 seconds) {noformat}
> {noformat}
> 0: jdbc:hive2://> show tables;
> +-+
> |tab_name |
> +-+
> | accommodations |
> | category|
> | date|
> | event |
> | listing |
> | sales |
> | sample |
> | test_float_4|
> | test_int_2 |
> | test_int_4 |
> | test_int_8 |
> | test_time |
> | test_time_2 |
> | test_timestamp |
> | users |
> | venue |
> | zipcode |
> +-+
> 17 rows selected (7.773 seconds) {noformat}
> {noformat}
> 0: jdbc:hive2://> select * from test_float_4;
> +-+
> | test_float_4.i |
> +-+
> | 1.0 |
> +-+
> 1 row selected (18.214 seconds) {noformat}
> {noformat}
> 0: jdbc:hive2://> select * from test_int_8;
> +---+
> | test_int_8.i |
> +---+
> | 1 |
> +---+
> 1 row selected (12.626 seconds) {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25826) Support table defaults at each database level
[ https://issues.apache.org/jira/browse/HIVE-25826?focusedWorklogId=714681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714681 ] ASF GitHub Bot logged work on HIVE-25826: - Author: ASF GitHub Bot Created on: 25/Jan/22 16:38 Start Date: 25/Jan/22 16:38 Worklog Time Spent: 10m Work Description: saihemanth-cloudera commented on a change in pull request #2924: URL: https://github.com/apache/hive/pull/2924#discussion_r791066005 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDatabaseTableDefault.java ## @@ -0,0 +1,304 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.ql; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.LocatedFileStatus; Review comment: Ack ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDatabaseTableDefault.java ## @@ -0,0 +1,304 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.ql; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.FileUtil; Review comment: Ack ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDatabaseTableDefault.java ## @@ -0,0 +1,304 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.ql; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem;
[jira] [Work logged] (HIVE-25893) NPE when reading Parquet data because ColumnVector isNull[] is not updated
[
https://issues.apache.org/jira/browse/HIVE-25893?focusedWorklogId=714657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714657
]
ASF GitHub Bot logged work on HIVE-25893:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 16:36
Start Date: 25/Jan/22 16:36
Worklog Time Spent: 10m
Work Description: soumyakanti3578 opened a new pull request #2970:
URL: https://github.com/apache/hive/pull/2970
Refactor `compareBytesColumnVector(BytesColumnVector cv1, BytesColumnVector
cv2)` for better readability, and copy `lcv.child.isNull` to `resultCV.isNull`
which was not getting updated earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714657)
Time Spent: 20m (was: 10m)
> NPE when reading Parquet data because ColumnVector isNull[] is not updated
> --
>
> Key: HIVE-25893
> URL: https://issues.apache.org/jira/browse/HIVE-25893
> Project: Hive
> Issue Type: Bug
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In
> [VectorizedListColumnReader.java|https://github.com/apache/hive/blob/595f3bc9d612f02581bd3377ee0107efd6553ae6/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java]
> {{isNull[]}} is used in the comparison methods ( eg.
> [columnVectorsDifferNullForSameIndex
> |https://github.com/apache/hive/blob/595f3bc9d612f02581bd3377ee0107efd6553ae6/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java#L524]
> ), however, {{isNull}} is always {{false}} as it is never updated in
> [getChildData|https://github.com/apache/hive/blob/595f3bc9d612f02581bd3377ee0107efd6553ae6/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java#L401].
> This could result in NullPointerException like,
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.compareBytesColumnVector(VectorizedListColumnReader.java:506)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.compareColumnVector(VectorizedListColumnReader.java:432)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.setIsRepeating(VectorizedListColumnReader.java:367)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.convertValueListToListColumnVector(VectorizedListColumnReader.java:360)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.readBatch(VectorizedListColumnReader.java:83)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedMapColumnReader.readBatch(VectorizedMapColumnReader.java:57)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:438)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:377)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:100)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:375)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25898) Compaction txn heartbeating after Worker timeout
[ https://issues.apache.org/jira/browse/HIVE-25898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh reassigned HIVE-25898: -- > Compaction txn heartbeating after Worker timeout > > > Key: HIVE-25898 > URL: https://issues.apache.org/jira/browse/HIVE-25898 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > > In some cases, when the compaction transaction is aborted, the hearbeater > thread is not shut down and keeps heartbeating. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25795) [CVE-2021-44228] Update log4j2 version to 2.15.0
[
https://issues.apache.org/jira/browse/HIVE-25795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481883#comment-17481883
]
Jota Martos commented on HIVE-25795:
Hi team,
I can see this issue was resolved but Hive 3.1.2 available
[here|https://archive.apache.org/dist/hive/] is still including an old version
of log4j, do you plan to update the current binary package or release a new
version of Hive with the new Log4j's libraries?
{code:java}
apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar
apache-hive-3.1.2-bin/lib/log4j-api-2.10.0.jar
apache-hive-3.1.2-bin/lib/log4j-core-2.10.0.jar
{code}
Thanks
> [CVE-2021-44228] Update log4j2 version to 2.15.0
>
>
> Key: HIVE-25795
> URL: https://issues.apache.org/jira/browse/HIVE-25795
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Nikhil Gupta
>Assignee: Jota Martos
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> [Worst Apache Log4j RCE Zero day Dropped on Internet - Cyber
> Kendra|https://www.cyberkendra.com/2021/12/worst-log4j-rce-zeroday-dropped-on.html]
> Vulnerability:
> https://github.com/apache/logging-log4j2/commit/7fe72d6
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25897) Move delta metric collection into AcidMetricsService
[ https://issues.apache.org/jira/browse/HIVE-25897?focusedWorklogId=714550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714550 ] ASF GitHub Bot logged work on HIVE-25897: - Author: ASF GitHub Bot Created on: 25/Jan/22 15:09 Start Date: 25/Jan/22 15:09 Worklog Time Spent: 10m Work Description: lcspinter opened a new pull request #2973: URL: https://github.com/apache/hive/pull/2973 ### What changes were proposed in this pull request? Refactor AcidMetricsService to collect delta metrics as well. ### Why are the changes needed? DeltaFilesMetricReporter and AcidMetricsService are two different threads collecting ACID related metrics. It makes sense to merge those threads since they share the same goal. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test, unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 714550) Remaining Estimate: 0h Time Spent: 10m > Move delta metric collection into AcidMetricsService > > > Key: HIVE-25897 > URL: https://issues.apache.org/jira/browse/HIVE-25897 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > DeltaFilesMetricReporter and AcidMetricsService are two different threads > collecting ACID related metrics. It makes sense to merge those threads since > they share the same goal. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25897) Move delta metric collection into AcidMetricsService
[ https://issues.apache.org/jira/browse/HIVE-25897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25897: -- Labels: pull-request-available (was: ) > Move delta metric collection into AcidMetricsService > > > Key: HIVE-25897 > URL: https://issues.apache.org/jira/browse/HIVE-25897 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > DeltaFilesMetricReporter and AcidMetricsService are two different threads > collecting ACID related metrics. It makes sense to merge those threads since > they share the same goal. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25795) [CVE-2021-44228] Update log4j2 version to 2.15.0
[ https://issues.apache.org/jira/browse/HIVE-25795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jota Martos reassigned HIVE-25795: -- Assignee: Jota Martos (was: Nikhil Gupta) > [CVE-2021-44228] Update log4j2 version to 2.15.0 > > > Key: HIVE-25795 > URL: https://issues.apache.org/jira/browse/HIVE-25795 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 3.1.2, 4.0.0 >Reporter: Nikhil Gupta >Assignee: Jota Martos >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > [Worst Apache Log4j RCE Zero day Dropped on Internet - Cyber > Kendra|https://www.cyberkendra.com/2021/12/worst-log4j-rce-zeroday-dropped-on.html] > Vulnerability: > https://github.com/apache/logging-log4j2/commit/7fe72d6 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25897) Move delta metric collection into AcidMetricsService
[ https://issues.apache.org/jira/browse/HIVE-25897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér reassigned HIVE-25897: > Move delta metric collection into AcidMetricsService > > > Key: HIVE-25897 > URL: https://issues.apache.org/jira/browse/HIVE-25897 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > > DeltaFilesMetricReporter and AcidMetricsService are two different threads > collecting ACID related metrics. It makes sense to merge those threads since > they share the same goal. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714479
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 13:51
Start Date: 25/Jan/22 13:51
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791730792
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: {}", ms);
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("Opening raw store with implementation class: {}",
rawStoreClassName);
+return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
HMSHandlerContext.getThreadId());
Review comment:
Got it, I opened https://issues.apache.org/jira/browse/HIVE-25896 for
the removal of `getThreadId`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714479)
Time Spent: 2h 40m (was: 2.5h)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25883) Enhance Compaction Cleaner to skip when there is nothing to do
[
https://issues.apache.org/jira/browse/HIVE-25883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481814#comment-17481814
]
Zoltan Haindrich commented on HIVE-25883:
-
{code}
// aborted txn: 3881
//
com:8020/warehouse/tablespace/managed/hive/test_835163/base_0003209_v0003877
//
com:8020/warehouse/tablespace/managed/hive/test_835163/delta_0003561_0003561_000
//
@,type:MAJOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,
// hasOldAbort:false,highestWriteId:3309,errorMessage:null
{code}
> Enhance Compaction Cleaner to skip when there is nothing to do
> --
>
> Key: HIVE-25883
> URL: https://issues.apache.org/jira/browse/HIVE-25883
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> the cleaner works the following way:
> * it identifies obsolete directories (delta dirs ; which doesn't have open
> txns)
> * removes them and done
> if there are no obsolete directoris that is attributed to that there might be
> open txns so the request should be retried later.
> however if for some reason the directory was already cleaned - similarily it
> has no obsolete directories; and thus the request is retried for forever
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714455=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714455
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 13:05
Start Date: 25/Jan/22 13:05
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791690315
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: {}", ms);
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("Opening raw store with implementation class: {}",
rawStoreClassName);
+return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
HMSHandlerContext.getThreadId());
Review comment:
Makes sense. Then we will be able to remove the ThreadId threadlocal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714455)
Time Spent: 2.5h (was: 2h 20m)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714441
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 12:48
Start Date: 25/Jan/22 12:48
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791676766
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: {}", ms);
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("Opening raw store with implementation class: {}",
rawStoreClassName);
+return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
HMSHandlerContext.getThreadId());
Review comment:
yes, i was thinking if we can remove the
[getThreadId](https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IHMSHandler.java#L43-L47)
from IHMSHandler, which is declared as `InterfaceAudience.Private`, the
`getThreadId` is only used for logging now, how about raising another pr for
this? I also want to improve the metrics and audit of the HMSHandler
https://github.com/apache/hive/pull/2441.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714441)
Time Spent: 2h 20m (was: 2h 10m)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714432
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 12:14
Start Date: 25/Jan/22 12:14
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791652057
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: {}", ms);
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("Opening raw store with implementation class: {}",
rawStoreClassName);
+return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
HMSHandlerContext.getThreadId());
Review comment:
I think the threadId is not used in the RawStoreProxy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714432)
Time Spent: 2h 10m (was: 2h)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714428
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 12:07
Start Date: 25/Jan/22 12:07
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791646932
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: " + ms + " from thread id: " +
HMSHandlerContext.getThreadId());
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("{}: Opening raw store with implementation class: {}",
HMSHandlerContext.getThreadId(), rawStoreClassName);
Review comment:
No, I have removed this method in this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714428)
Time Spent: 2h (was: 1h 50m)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714423=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714423
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 12:00
Start Date: 25/Jan/22 12:00
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791642104
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: " + ms + " from thread id: " +
HMSHandlerContext.getThreadId());
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("{}: Opening raw store with implementation class: {}",
HMSHandlerContext.getThreadId(), rawStoreClassName);
Review comment:
Is `addPrefix` used anywhere in the code after this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714423)
Time Spent: 1h 50m (was: 1h 40m)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714414
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 11:45
Start Date: 25/Jan/22 11:45
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791630926
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: " + ms + " from thread id: " +
HMSHandlerContext.getThreadId());
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("{}: Opening raw store with implementation class: {}",
HMSHandlerContext.getThreadId(), rawStoreClassName);
Review comment:
The `addPrefix` method will insert the threadId before the log, unsure
why do like this. It makes sense for me to remove the threadId from the log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714414)
Time Spent: 1h 40m (was: 1.5h)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714408
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 11:29
Start Date: 25/Jan/22 11:29
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791618884
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -705,32 +635,22 @@ public static RawStore getMSForConf(Configuration conf)
throws MetaException {
ms.shutdown();
throw e;
}
- threadLocalMS.set(ms);
- ms = threadLocalMS.get();
- LOG.info("Created RawStore: " + ms + " from thread id: " +
Thread.currentThread().getId());
+ HMSHandlerContext.setRawStore(ms);
+ LOG.info("Created RawStore: " + ms + " from thread id: " +
HMSHandlerContext.getThreadId());
}
return ms;
}
@Override
public TxnStore getTxnHandler() {
-return getMsThreadTxnHandler(conf);
- }
-
- public static TxnStore getMsThreadTxnHandler(Configuration conf) {
-TxnStore txn = threadLocalTxn.get();
-if (txn == null) {
- txn = TxnUtils.getTxnStore(conf);
- threadLocalTxn.set(txn);
-}
-return txn;
+return HMSHandlerContext.getTxnStore(conf);
}
static RawStore newRawStoreForConf(Configuration conf) throws MetaException {
Configuration newConf = new Configuration(conf);
String rawStoreClassName = MetastoreConf.getVar(newConf,
ConfVars.RAW_STORE_IMPL);
-LOG.info(addPrefix("Opening raw store with implementation class:" +
rawStoreClassName));
-return RawStoreProxy.getProxy(newConf, conf, rawStoreClassName,
threadLocalId.get());
+LOG.info("{}: Opening raw store with implementation class: {}",
HMSHandlerContext.getThreadId(), rawStoreClassName);
Review comment:
Adding `threadId` to the log might not be needed. Our stack is
configured so the logger automatically logs the threadId. This should be so in
every environment where the threading is a serious issue, so I think this is
not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714408)
Time Spent: 1.5h (was: 1h 20m)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714394
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 11:02
Start Date: 25/Jan/22 11:02
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791598568
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandlerContext.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+
+/**
+ * When one hms client connects in, we create a handler context for it.
+ * We store session information here.
+ */
+public final class HMSHandlerContext {
+
+ private static final ThreadLocal context = new
ThreadLocal<>();
+
+ private static final AtomicInteger nextSerialNum = new AtomicInteger();
+
+ private RawStore rawStore;
+
+ private TxnStore txnStore;
+
+ // Thread local HMSHandler used during shutdown to notify meta listeners
+ private HMSHandler hmsHandler;
+
+ // Thread local configuration is needed as many threads could make changes
+ // to the conf using the connection hook
+ private Configuration configuration;
+
+ // Thread local Map to keep track of modified meta conf keys
+ private Map modifiedConfig = new HashMap<>();
+
+ private Integer threadId = nextSerialNum.incrementAndGet();
+ // This will only be set if the metastore is being accessed from a metastore
Thrift server,
+ // not if it is from the CLI. Also, only if the TTransport being used to
connect is an
+ // instance of TSocket. This is also not set when kerberos is used.
+ private String ipAddress;
+
+ private Map timerContexts = new
HashMap<>();
+
+ private HMSHandlerContext() {
+
+ }
+
+ public static Optional getRawStore() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalRawStore() : Optional.empty();
+ }
+
+ public static Optional getHMSHandler() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalHmsHandler() : Optional.empty();
+ }
+
+ public static Optional getIpAddress() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getRemoteIpAddress() : Optional.empty();
+ }
+
+ public static Optional getConfiguration() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalConfiguration() : Optional.empty();
+ }
+
+ public static TxnStore getTxnStore(Configuration conf) {
+return getContext().getLocalTxnStore().orElseGet(() -> {
+ TxnStore txnStore = TxnUtils.getTxnStore(conf);
+ setTxnStore(txnStore);
+ return txnStore;
+});
+ }
+
+ public static Map getModifiedConfig() {
+return getContext().modifiedConfig;
+ }
+
+ public static Integer getThreadId() {
+return getContext().threadId;
+ }
+
+ public static Map
getTimerContexts() {
+return getContext().timerContexts;
+ }
+
+ private static HMSHandlerContext getContext() {
+HMSHandlerContext ctx = context.get();
+if (ctx == null) {
+ context.set(ctx = new HMSHandlerContext());
+}
+return ctx;
+ }
+
+ public static void setRawStore(RawStore rawStore) {
+getContext().rawStore = rawStore;
+ }
+
+ public static void setTxnStore(TxnStore txnStore) {
+getContext().txnStore = txnStore;
+ }
+
+ public static void setHMSHandler(HMSHandler hmsHandler) {
+getContext().hmsHandler = hmsHandler;
+ }
+
+ public static void setConfiguration(Configuration conf) {
+getContext().configuration = conf;
+ }
+
+ public static void setIpAddress(String ipAddress) {
+getContext().ipAddress = ipAddress;
+ }
+
+ public static
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714386
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 10:56
Start Date: 25/Jan/22 10:56
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791589935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -169,65 +169,29 @@
private Warehouse wh; // hdfs warehouse
private static Striped tablelocks;
- private static final ThreadLocal threadLocalMS = new
ThreadLocal();
- private static final ThreadLocal threadLocalTxn = new
ThreadLocal();
-
- private static final ThreadLocal> timerContexts =
- new ThreadLocal>() {
-@Override
-protected Map
initialValue() {
- return new HashMap<>();
-}
- };
-
public static RawStore getRawStore() {
-return threadLocalMS.get();
+return HMSHandlerContext.getRawStore().orElse(null);
}
static void cleanupRawStore() {
Review comment:
It's been used only in two places(HMSHandler) when client
disconnects，can we rename it to `cleanupHandlerContext` with public modifier?
Thank you for the suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714386)
Time Spent: 1h 10m (was: 1h)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714378
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 10:52
Start Date: 25/Jan/22 10:52
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791589935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -169,65 +169,29 @@
private Warehouse wh; // hdfs warehouse
private static Striped tablelocks;
- private static final ThreadLocal threadLocalMS = new
ThreadLocal();
- private static final ThreadLocal threadLocalTxn = new
ThreadLocal();
-
- private static final ThreadLocal> timerContexts =
- new ThreadLocal>() {
-@Override
-protected Map
initialValue() {
- return new HashMap<>();
-}
- };
-
public static RawStore getRawStore() {
-return threadLocalMS.get();
+return HMSHandlerContext.getRawStore().orElse(null);
}
static void cleanupRawStore() {
Review comment:
It's been used only in two places(HMSHandler) when client
disconnects，can we rename it to `cleanupContext` with public modifier? Thank
you for the suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714378)
Time Spent: 1h (was: 50m)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714375=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714375
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 10:47
Start Date: 25/Jan/22 10:47
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791585817
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandlerContext.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+
+/**
+ * When one hms client connects in, we create a handler context for it.
+ * We store session information here.
+ */
+public final class HMSHandlerContext {
+
+ private static final ThreadLocal context = new
ThreadLocal<>();
+
+ private static final AtomicInteger nextSerialNum = new AtomicInteger();
+
+ private RawStore rawStore;
+
+ private TxnStore txnStore;
+
+ // Thread local HMSHandler used during shutdown to notify meta listeners
+ private HMSHandler hmsHandler;
+
+ // Thread local configuration is needed as many threads could make changes
+ // to the conf using the connection hook
+ private Configuration configuration;
+
+ // Thread local Map to keep track of modified meta conf keys
+ private Map modifiedConfig = new HashMap<>();
+
+ private Integer threadId = nextSerialNum.incrementAndGet();
+ // This will only be set if the metastore is being accessed from a metastore
Thrift server,
+ // not if it is from the CLI. Also, only if the TTransport being used to
connect is an
+ // instance of TSocket. This is also not set when kerberos is used.
+ private String ipAddress;
+
+ private Map timerContexts = new
HashMap<>();
+
+ private HMSHandlerContext() {
+
+ }
+
+ public static Optional getRawStore() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalRawStore() : Optional.empty();
+ }
+
+ public static Optional getHMSHandler() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalHmsHandler() : Optional.empty();
+ }
+
+ public static Optional getIpAddress() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getRemoteIpAddress() : Optional.empty();
+ }
+
+ public static Optional getConfiguration() {
+HMSHandlerContext ctx = context.get();
+return ctx != null ? ctx.getLocalConfiguration() : Optional.empty();
+ }
+
+ public static TxnStore getTxnStore(Configuration conf) {
+return getContext().getLocalTxnStore().orElseGet(() -> {
+ TxnStore txnStore = TxnUtils.getTxnStore(conf);
+ setTxnStore(txnStore);
+ return txnStore;
+});
+ }
+
+ public static Map getModifiedConfig() {
+return getContext().modifiedConfig;
+ }
+
+ public static Integer getThreadId() {
+return getContext().threadId;
+ }
+
+ public static Map
getTimerContexts() {
+return getContext().timerContexts;
+ }
+
+ private static HMSHandlerContext getContext() {
+HMSHandlerContext ctx = context.get();
+if (ctx == null) {
+ context.set(ctx = new HMSHandlerContext());
+}
+return ctx;
+ }
+
+ public static void setRawStore(RawStore rawStore) {
+getContext().rawStore = rawStore;
+ }
+
+ public static void setTxnStore(TxnStore txnStore) {
+getContext().txnStore = txnStore;
+ }
+
+ public static void setHMSHandler(HMSHandler hmsHandler) {
+getContext().hmsHandler = hmsHandler;
+ }
+
+ public static void setConfiguration(Configuration conf) {
+getContext().configuration = conf;
+ }
+
+ public static void setIpAddress(String ipAddress) {
+getContext().ipAddress = ipAddress;
+ }
+
+ public static void
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[
https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714373
]
ASF GitHub Bot logged work on HIVE-25892:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 10:42
Start Date: 25/Jan/22 10:42
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2967:
URL: https://github.com/apache/hive/pull/2967#discussion_r791581995
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -169,65 +169,29 @@
private Warehouse wh; // hdfs warehouse
private static Striped tablelocks;
- private static final ThreadLocal threadLocalMS = new
ThreadLocal();
- private static final ThreadLocal threadLocalTxn = new
ThreadLocal();
-
- private static final ThreadLocal> timerContexts =
- new ThreadLocal>() {
-@Override
-protected Map
initialValue() {
- return new HashMap<>();
-}
- };
-
public static RawStore getRawStore() {
-return threadLocalMS.get();
+return HMSHandlerContext.getRawStore().orElse(null);
}
static void cleanupRawStore() {
Review comment:
This is not a `cleanupRawStore` for a good while...
How widely used is this? Could we rename this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714373)
Time Spent: 40m (was: 0.5h)
> Group HMSHandler's thread locals into a single context
> --
>
> Key: HIVE-25892
> URL: https://issues.apache.org/jira/browse/HIVE-25892
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> There are more than six ThreadLocal variables in HMSHandler, we can group
> them together into a single context to improve the management of variables
> and the code readability.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work started] (HIVE-25746) Compaction Failure Counter counted incorrectly
[
https://issues.apache.org/jira/browse/HIVE-25746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-25746 started by Viktor Csomor.
> Compaction Failure Counter counted incorrectly
> --
>
> Key: HIVE-25746
> URL: https://issues.apache.org/jira/browse/HIVE-25746
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>
> The count of the below metrics counted incorrectly upon an exception.
> - {{compaction_initator_failure_counter}}
> - {{compaction_cleaner_failure_counter}}
> Reasoning:
> In the {{Initator}}/{{Cleaner}} class creates a list of {{CompletableFuture}}
> which {{Runnable}} core exception is being wrapped to {{RuntimeExceptions}}.
> The below code-snippet waits all cleaners to complete (Initiators does it
> similarly).
> {code:java}
> try {
>
> for (CompactionInfo compactionInfo : readyToClean) {
>
> cleanerList.add(CompletableFuture.runAsync(CompactorUtil.ThrowingRunnable.unchecked(()
> ->
> clean(compactionInfo, cleanerWaterMark,
> metricsEnabled)), cleanerExecutor));
> }
> CompletableFuture.allOf(cleanerList.toArray(new
> CompletableFuture[0])).join();
> }
> } catch (Throwable t) {
> // the lock timeout on AUX lock, should be ignored.
> if (metricsEnabled && handle != null) {
> failuresCounter.inc();
> }
> {code}
> If the {{CompleteableFututre#join}} throws an Exception then the failure
> counter is incremented.
> Docs:
> {code}
> /**
> * Returns the result value when complete, or throws an
> * (unchecked) exception if completed exceptionally. To better
> * conform with the use of common functional forms, if a
> * computation involved in the completion of this
> * CompletableFuture threw an exception, this method throws an
> * (unchecked) {@link CompletionException} with the underlying
> * exception as its cause.
> *
> * @return the result value
> * @throws CancellationException if the computation was cancelled
> * @throws CompletionException if this future completed
> * exceptionally or a completion computation threw an exception
> */
> public T join() {
> Object r;
> return reportJoin((r = result) == null ? waitingGet(false) : r);
> }
> {code}
> (!) Let's suppose we have 10 cleaners and the 2nd throws an exception. The
> {{catch}} block will be initiated and the {{failuresCounter}} will be
> incremented. If there is any consecutive error amongst the remaining cleaners
> the counter won't be incremented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25706) ShuffleHandler: Make sure of properly releasing netty reference counted objects
[
https://issues.apache.org/jira/browse/HIVE-25706?focusedWorklogId=714366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714366
]
ASF GitHub Bot logged work on HIVE-25706:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 10:24
Start Date: 25/Jan/22 10:24
Worklog Time Spent: 10m
Work Description: abstractdog opened a new pull request #2794:
URL: https://github.com/apache/hive/pull/2794
### What changes were proposed in this pull request?
Release the ByteBuf instance at the last usage point instead of releasing
the http response.
### Why are the changes needed?
It threw exceptions silently in ShuffleHandler logs, easily reproducible in
TestShuffleHandler.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Rerun TestShuffleHandler, IllegalReferenceCountException disappeared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714366)
Time Spent: 40m (was: 0.5h)
> ShuffleHandler: Make sure of properly releasing netty reference counted
> objects
> ---
>
> Key: HIVE-25706
> URL: https://issues.apache.org/jira/browse/HIVE-25706
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> https://netty.io/wiki/reference-counted-objects.html
> ShuffleHandler log contains issues below when we're on unhappy codepath
> (sendError). Very easily reproducible e.g. with calling ShuffleHandler's http
> port without any parameters. Doesn't cause a user-facing issue, because at
> this point ShuffleHandler already responded to the client, but the exception
> is annoying and implies invalid .release() calls in netty4.
> {code}
> io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
> at
> io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.handler.codec.http.DefaultFullHttpResponse.release(DefaultFullHttpResponse.java:116)
> ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.sendError(ShuffleHandler.java:1056)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.handleRequest(ShuffleHandler.java:827)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.channelRead(ShuffleHandler.java:728)
> ~[classes/:?]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-all-4.1.65.Final.jar:4.1.65.Final]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25894) Table migration to Iceberg doesn't remove HMS partitions
[
https://issues.apache.org/jira/browse/HIVE-25894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod reassigned HIVE-25894:
-
Assignee: Marton Bod
> Table migration to Iceberg doesn't remove HMS partitions
>
>
> Key: HIVE-25894
> URL: https://issues.apache.org/jira/browse/HIVE-25894
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Marton Bod
>Priority: Major
>
> Repro:
> {code:java}
> create table ice_part_migrate (i int) partitioned by (p int) stored as
> parquet;
> insert into ice_part_migrate partition(p=1) values (1), (11), (111);
> insert into ice_part_migrate partition(p=2) values (2), (22), (222);
> ALTER TABLE ice_part_migrate SET TBLPROPERTIES
> ('storage_handler'='org.apache.iceberg.mr.hive.HiveIcebergStorageHandler');
> {code}
> Then looking at the HMS database:
> {code:java}
> => select "PART_NAME" from "PARTITIONS" p, "TBLS" t where
> t."TBL_ID"=p."TBL_ID" and t."TBL_NAME"='ice_part_migrate';
> PART_NAME
> ---
> p=1
> p=2
> {code}
> This is weird because Iceberg tables are supposed to be unpartitioned. It
> also breaks some precondition checks in Impala. Is there a particular reason
> to keep the partitions in HMS?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25879) MetaStoreDirectSql test query should not query the DBS table
[
https://issues.apache.org/jira/browse/HIVE-25879?focusedWorklogId=714345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714345
]
ASF GitHub Bot logged work on HIVE-25879:
-
Author: ASF GitHub Bot
Created on: 25/Jan/22 09:53
Start Date: 25/Jan/22 09:53
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #2964:
URL: https://github.com/apache/hive/pull/2964#issuecomment-1021002371
@mszurap: Could you please separate the 2 changes?
Could you please make sure that the tests are green?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 714345)
Time Spent: 20m (was: 10m)
> MetaStoreDirectSql test query should not query the DBS table
>
>
> Key: HIVE-25879
> URL: https://issues.apache.org/jira/browse/HIVE-25879
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Szurap
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The runTestQuery() in the
> org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java is using a test query
> {code:java}
> select "DB_ID" from "DBS"{code}
> to determine whether the direct SQL can be used.
> With larger deployments with many (10k+) Hive databases it would be more
> efficienct to query a small table instead, for example the "VERSION" table
> should always have a single row only.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25894) Table migration to Iceberg doesn't remove HMS partitions
[
https://issues.apache.org/jira/browse/HIVE-25894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481685#comment-17481685
]
Zoltán Borók-Nagy commented on HIVE-25894:
--
Yeah, in Impala we have a check that if partition keys are empty, then there
shouldn't be any partitions:
[https://github.com/apache/impala/blob/12118664d84599f4164494058d35a882ba2dd193/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L685-L686]
I can change our preconditions, but if you also consider this to be a bug, then
I think it's better to fix at the Hive side.
> Table migration to Iceberg doesn't remove HMS partitions
>
>
> Key: HIVE-25894
> URL: https://issues.apache.org/jira/browse/HIVE-25894
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Priority: Major
>
> Repro:
> {code:java}
> create table ice_part_migrate (i int) partitioned by (p int) stored as
> parquet;
> insert into ice_part_migrate partition(p=1) values (1), (11), (111);
> insert into ice_part_migrate partition(p=2) values (2), (22), (222);
> ALTER TABLE ice_part_migrate SET TBLPROPERTIES
> ('storage_handler'='org.apache.iceberg.mr.hive.HiveIcebergStorageHandler');
> {code}
> Then looking at the HMS database:
> {code:java}
> => select "PART_NAME" from "PARTITIONS" p, "TBLS" t where
> t."TBL_ID"=p."TBL_ID" and t."TBL_NAME"='ice_part_migrate';
> PART_NAME
> ---
> p=1
> p=2
> {code}
> This is weird because Iceberg tables are supposed to be unpartitioned. It
> also breaks some precondition checks in Impala. Is there a particular reason
> to keep the partitions in HMS?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25894) Table migration to Iceberg doesn't remove HMS partitions
[
https://issues.apache.org/jira/browse/HIVE-25894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481669#comment-17481669
]
Marton Bod commented on HIVE-25894:
---
[~boroknagyz] interesting, thanks for raising this! If you load the table from
HMS, it is unpartitioned:
table.getPartitionKeys() gives back an empty list. Not sure why the partitions
are not purged from the database too and whether it causes any problems.
> Table migration to Iceberg doesn't remove HMS partitions
>
>
> Key: HIVE-25894
> URL: https://issues.apache.org/jira/browse/HIVE-25894
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Priority: Major
>
> Repro:
> {code:java}
> create table ice_part_migrate (i int) partitioned by (p int) stored as
> parquet;
> insert into ice_part_migrate partition(p=1) values (1), (11), (111);
> insert into ice_part_migrate partition(p=2) values (2), (22), (222);
> ALTER TABLE ice_part_migrate SET TBLPROPERTIES
> ('storage_handler'='org.apache.iceberg.mr.hive.HiveIcebergStorageHandler');
> {code}
> Then looking at the HMS database:
> {code:java}
> => select "PART_NAME" from "PARTITIONS" p, "TBLS" t where
> t."TBL_ID"=p."TBL_ID" and t."TBL_NAME"='ice_part_migrate';
> PART_NAME
> ---
> p=1
> p=2
> {code}
> This is weird because Iceberg tables are supposed to be unpartitioned. It
> also breaks some precondition checks in Impala. Is there a particular reason
> to keep the partitions in HMS?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25892) Group HMSHandler's thread locals into a single context
[ https://issues.apache.org/jira/browse/HIVE-25892?focusedWorklogId=714277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-714277 ] ASF GitHub Bot logged work on HIVE-25892: - Author: ASF GitHub Bot Created on: 25/Jan/22 08:26 Start Date: 25/Jan/22 08:26 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2967: URL: https://github.com/apache/hive/pull/2967#issuecomment-1020924354 > Could we provide a way to clear the context? In Iceberg we create a HMS instance for our tests, and because of the threadlocals here and in the TxnHandler we leak information between the tests. Add a [CleanupHook](https://github.com/apache/hive/blob/c8c6152b79810276ac070ec057b0643b9ce3b03f/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandlerContext.java#L170-L172) here to clear the locals by calling HMSHandlerContext#clear(CleanupHook cleanupHook), not sure it enough for the iceberg case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 714277) Time Spent: 0.5h (was: 20m) > Group HMSHandler's thread locals into a single context > -- > > Key: HIVE-25892 > URL: https://issues.apache.org/jira/browse/HIVE-25892 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > There are more than six ThreadLocal variables in HMSHandler, we can group > them together into a single context to improve the management of variables > and the code readability. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
