[jira] [Updated] (HIVE-28109) Hive Metastore lock ignores catalog
[ https://issues.apache.org/jira/browse/HIVE-28109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-28109: - Parent: HIVE-18685 Issue Type: Sub-task (was: Bug) > Hive Metastore lock ignores catalog > --- > > Key: HIVE-28109 > URL: https://issues.apache.org/jira/browse/HIVE-28109 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Szehon Ho >Priority: Major > > HIVE-18685 added catalogs to Hive. > > However, IMetastoreClient::lock ignores catalog. There is thus no way to > lock a table of one catalog, if a table of the same db/tableName in another > catalog is locked. We should add catName to the IMetastoreClient::lock API, > in par with the other API's. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28109) Hive Metastore lock ignores catalog
Szehon Ho created HIVE-28109: Summary: Hive Metastore lock ignores catalog Key: HIVE-28109 URL: https://issues.apache.org/jira/browse/HIVE-28109 Project: Hive Issue Type: Bug Components: Standalone Metastore Reporter: Szehon Ho HIVE-18685 added catalogs to Hive. However, IMetastoreClient::lock ignores catalog. There is thus no way to lock a table of one catalog, if a table of the same db/tableName in another catalog is locked. We should add catName to the IMetastoreClient::lock API, in par with the other API's. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Fix Version/s: 3.1.3
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.3, 4.0.0
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.ja
[jira] [Commented] (HIVE-25855) Make a branch-3 release
[ https://issues.apache.org/jira/browse/HIVE-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503147#comment-17503147 ] Szehon Ho commented on HIVE-25855: -- Hi [~ngangam], the JIRA has been backported (via HIVE-25991), if it can still be included. Thanks > Make a branch-3 release > > > Key: HIVE-25855 > URL: https://issues.apache.org/jira/browse/HIVE-25855 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > This jira is to track commits for a hive release off branch-3 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25991) Backport HIVE-25522 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-25991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho resolved HIVE-25991. -- Fix Version/s: 3.2.0 Resolution: Fixed > Backport HIVE-25522 to branch-3 > --- > > Key: HIVE-25991 > URL: https://issues.apache.org/jira/browse/HIVE-25991 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.2 >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25991) Backport HIVE-25522 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-25991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499176#comment-17499176 ] Szehon Ho commented on HIVE-25991: -- [~ngangam] made this backport PR, let me know if it looks right. Will monitor tests > Backport HIVE-25522 to branch-3 > --- > > Key: HIVE-25991 > URL: https://issues.apache.org/jira/browse/HIVE-25991 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.2 >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25991) Backport HIVE-25522 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-25991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho reassigned HIVE-25991: Assignee: Szehon Ho > Backport HIVE-25522 to branch-3 > --- > > Key: HIVE-25991 > URL: https://issues.apache.org/jira/browse/HIVE-25991 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.2 >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25855) Make a branch-3 release
[ https://issues.apache.org/jira/browse/HIVE-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498552#comment-17498552 ] Szehon Ho commented on HIVE-25855: -- Hi [~ngangam] thanks for volunteering for this. May i try to cherry-pick HIVE-25522 (I hit this in Hive 3 actually..), and how can I do this if so? Thanks again > Make a branch-3 release > > > Key: HIVE-25855 > URL: https://issues.apache.org/jira/browse/HIVE-25855 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > This jira is to track commits for a hive release off branch-3 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho resolved HIVE-25522.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(T
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426417#comment-17426417
]
Szehon Ho edited comment on HIVE-25522 at 10/8/21, 11:46 PM:
-
Update, as I posted on the PR, realized its not affecting current master branch
with default settings but only 3.x, because of new added AcidMetricsService
which is initialized eagerly and will fail HMS rather than leave the static
TxnHandler fields in the corrupt state.
So removing 4.x from affected issues (though it might be an issue if
AcidMetricsService is explicitly disabled).
was (Author: szehon):
Update, as I posted on the PR, realized its not affecting current master branch
with default settings but only 3.x, because of new added AcidMetricsService
which is initialized eagerly and will fail HMS rather than leave the static
TxnHandler fields in the corrupt state. So taking it off of affected issues
(though it might be an issue if AcidMetricsService is explicitly disabled).
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
>
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426417#comment-17426417
]
Szehon Ho commented on HIVE-25522:
--
Update, as I posted on the PR, realized its not affecting current master branch
with default settings but only 3.x, because of new added AcidMetricsService
which is initialized eagerly and will fail HMS rather than leave the static
TxnHandler fields in the corrupt state. So taking it off of affected issues
(though it might be an issue if AcidMetricsService is explicitly disabled).
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastor
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Affects Version/s: (was: 4.0.0)
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.
[jira] [Resolved] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho resolved HIVE-24263. -- Resolution: Duplicate > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24263.patch > > Time Spent: 40m > Remaining Estimate: 0h > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24263 started by Szehon Ho. > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24263.patch > > Time Spent: 40m > Remaining Estimate: 0h > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-24263: - Status: Open (was: Patch Available) > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24263.patch > > Time Spent: 40m > Remaining Estimate: 0h > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20789) HiveServer2 should have Timeouts against clients that never close sockets
[
https://issues.apache.org/jira/browse/HIVE-20789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20789:
-
Status: Open (was: Patch Available)
> HiveServer2 should have Timeouts against clients that never close sockets
> -
>
> Key: HIVE-20789
> URL: https://issues.apache.org/jira/browse/HIVE-20789
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-20789.2.patch, HIVE-20789.patch
>
>
> We have had a scenario that health checks sending 0 bytes to HiveServer2
> sockets would DDOS the HiveServer2, if for some reason they hang or otherwise
> don't send TCP FIN, then all HiveServer2 thrift thread-pool threads will
> block reading the socket.
> This is the stack (we are running an older version of Hive here)
> {noformat}
> "HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <23781b74> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Eventually HiveServer2 has no more free threads left.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-20789) HiveServer2 should have Timeouts against clients that never close sockets
[
https://issues.apache.org/jira/browse/HIVE-20789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho resolved HIVE-20789.
--
Resolution: Won't Fix
> HiveServer2 should have Timeouts against clients that never close sockets
> -
>
> Key: HIVE-20789
> URL: https://issues.apache.org/jira/browse/HIVE-20789
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-20789.2.patch, HIVE-20789.patch
>
>
> We have had a scenario that health checks sending 0 bytes to HiveServer2
> sockets would DDOS the HiveServer2, if for some reason they hang or otherwise
> don't send TCP FIN, then all HiveServer2 thrift thread-pool threads will
> block reading the socket.
> This is the stack (we are running an older version of Hive here)
> {noformat}
> "HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <23781b74> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Eventually HiveServer2 has no more free threads left.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415278#comment-17415278
]
Szehon Ho edited comment on HIVE-25522 at 9/15/21, 4:42 PM:
[~csun] helped me to reason through the code and pointed out the potential
error, thanks!
It looks like the TxnHandler's initialization block sets a lot of static
variables llke connPool, sqlGenerator, etc.. But it skips it if only connPool
is set. So it's possible that connPool was successfully set but the thread
crashed and did not set the other variables. Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L372
It looks like TxnHandler's are thread local and are constructed on demand, so a
fatal error in initialization does not kill the HMS server.
was (Author: szehon):
[~csun] helped me to reason through the code and pointed out the potential
error, thanks!
It looks like the TxnHandler's initialization block sets a lot of static
variables llke connPool, sqlGenerator, etc.. But it skips it if only connPool
is set. So it's possible that connPool was successfully set but the thread
crashed and did not set the other variables. Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L372
It looks like TxnHandler's are thread local and are constructed on demand, so a
fatal error in one thread does not kill the HMS server.
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.j
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415625#comment-17415625
]
Szehon Ho edited comment on HIVE-25522 at 9/15/21, 4:41 PM:
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after connPool is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
was (Author: szehon):
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after connPool
is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHa
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415625#comment-17415625
]
Szehon Ho edited comment on HIVE-25522 at 9/15/21, 4:41 PM:
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after connPool
is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
was (Author: szehon):
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after dbConn is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHand
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Affects Version/s: 4.0.0
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
> at j
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415625#comment-17415625
]
Szehon Ho commented on HIVE-25522:
--
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after dbConn is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.proce
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415624#comment-17415624
]
Szehon Ho commented on HIVE-25522:
--
Hi [~pvary] thanks for the reply, I found the issue last night, can give a stab
at a pr today. Thanks
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.1.2.jar:3.1
[jira] [Assigned] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-25522:
Assignee: Szehon Ho
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
> at javax.
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415278#comment-17415278
]
Szehon Ho edited comment on HIVE-25522 at 9/15/21, 1:18 AM:
[~csun] helped me to reason through the code and pointed out the potential
error, thanks!
It looks like the TxnHandler's initialization block sets a lot of static
variables llke connPool, sqlGenerator, etc.. But it skips it if only connPool
is set. So it's possible that connPool was successfully set but the thread
crashed and did not set the other variables. Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L372
It looks like TxnHandler's are thread local and are constructed on demand, so a
fatal error in one thread does not kill the HMS server.
was (Author: szehon):
[~csun] helped me to reason through the code, thanks!
It looks like the TxnHandler's initialization block sets a lot of static
variables llke connPool, sqlGenerator, etc.. But it skips it if only connPool
is set. So it's possible that connPool was successfully set but the thread
crashed and did not set the other variables. Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L372
It looks like TxnHandler's are thread local and are constructed on demand, so a
fatal error in one thread does not kill the HMS server.
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.Generat
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415278#comment-17415278
]
Szehon Ho commented on HIVE-25522:
--
[~csun] helped me to reason through the code, thanks!
It looks like the TxnHandler's initialization block sets a lot of static
variables llke connPool, sqlGenerator, etc.. But it skips it if only connPool
is set. So it's possible that connPool was successfully set but the thread
crashed and did not set the other variables. Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L372
It looks like TxnHandler's are thread local and are constructed on demand, so a
fatal error in one thread does not kill the HMS server.
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
>
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Description:
Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg issues
a lot of lock() calls for commits.
We hit randomly a strange NPE that fails Iceberg commits.
{noformat}
2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] metastore.RetryingHMSHandler:
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy27.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
Error occurred during processing of message.
java.lang.NullPointerException: null
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
~[hive-exec-3.1.2.jar:3.1.2]
at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
~[hive-exec-3.1.2.jar:3.1.2]
at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
~[hive-exec-3.1.2.jar:3.1.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
~[hadoop-common-3.1.4.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-3.1.2.jar:3.1.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Description:
Environment: Using Iceberg on Hive (which uses a lot of lock() calls)
Hit a very strange NPE:
{noformat}
2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] metastore.RetryingHMSHandler:
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy27.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
Error occurred during processing of message.
java.lang.NullPointerException: null
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
~[hive-exec-3.1.2.jar:3.1.2]
at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
~[hive-exec-3.1.2.jar:3.1.2]
at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
~[hive-exec-3.1.2.jar:3.1.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
~[hadoop-common-3.1.4.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-3.1.2.jar:3.1.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Description:
Environment: Using Iceberg on Hive (which uses a lot of lock() calls)
Hit a very strange NPE:
{noformat}
2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] metastore.RetryingHMSHandler:
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy27.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
Error occurred during processing of message.
java.lang.NullPointerException: null
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
~[hive-exec-3.1.2.jar:3.1.2]
at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
~[hive-exec-3.1.2.jar:3.1.2]
at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
~[hive-exec-3.1.2.jar:3.1.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
~[hadoop-common-3.1.4.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
~[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-3.1.2.jar:3.1.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker
[jira] [Commented] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217081#comment-17217081 ] Szehon Ho commented on HIVE-24263: -- Ah I did not see this, will take a look, thanks for the pointer > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24263.patch > > Time Spent: 20m > Remaining Estimate: 0h > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212344#comment-17212344 ] Szehon Ho commented on HIVE-24263: -- I made a PR to show how this could work, would love to get some feedback > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24263.patch > > Time Spent: 10m > Remaining Estimate: 0h > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-24263: - Status: Patch Available (was: Open) > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24263.patch > > Time Spent: 10m > Remaining Estimate: 0h > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-24263: - Description: In our company, we have a use-case to get quickly a list of partition locations. Currently it is done via listPartitions, which is a very heavy operation in terms of memory and performance. This JIRA proposes an API: Map listPartitionLocations(String db, String table, short max) that returns a map of partition names to locations. For example, we have an integration from output of a Hive pipeline to Spark jobs that consume directly from HDFS. The Spark job scheduler needs to know the partition paths that are available for consumption (the partition name is not sufficient as it's input is HDFS path), and so we have to do heavy listPartitions() for this. Another use-case is for a HDFS data removal tool that does a nightly crawl to see if there are associated hive partitions mapped to a given partition path. The nightly crawling job could be much less resource-intensive if we had a listPartitionLocations(). As there is already an internal method in the ObjectStore for this done for dropPartitions, it is only a matter of exposing this API to HiveMetaStoreClient. was: In our company, we have a use-case to get quickly a list of partition locations. Currently it is done via listPartitions, which is a very heavy operation in terms of memory and performance. For example, we have an integration from output of a Hive pipeline to Spark jobs that consume directly from HDFS. It needs to know the partition paths that are available for consumation, and does repeated listPartitions() for this. As there is already an internal method in the ObjectStore for this done for dropPartitions, it is only a matter of exposing this API to HiveMetaStoreClient. > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Priority: Major > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-24263: - Attachment: HIVE-24263.patch > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-24263.patch > > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24263) Create an HMS endpoint to list partition locations
[ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho reassigned HIVE-24263: Assignee: Szehon Ho > Create an HMS endpoint to list partition locations > -- > > Key: HIVE-24263 > URL: https://issues.apache.org/jira/browse/HIVE-24263 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > > In our company, we have a use-case to get quickly a list of partition > locations. Currently it is done via listPartitions, which is a very heavy > operation in terms of memory and performance. > This JIRA proposes an API: Map listPartitionLocations(String > db, String table, short max) that returns a map of partition names to > locations. > For example, we have an integration from output of a Hive pipeline to Spark > jobs that consume directly from HDFS. The Spark job scheduler needs to know > the partition paths that are available for consumption (the partition name is > not sufficient as it's input is HDFS path), and so we have to do heavy > listPartitions() for this. > Another use-case is for a HDFS data removal tool that does a nightly crawl to > see if there are associated hive partitions mapped to a given partition path. > The nightly crawling job could be much less resource-intensive if we had a > listPartitionLocations(). > As there is already an internal method in the ObjectStore for this done for > dropPartitions, it is only a matter of exposing this API to > HiveMetaStoreClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17206064#comment-17206064
]
Szehon Ho commented on HIVE-19253:
--
i think there's no new test failures [~vihangk1] what do you think about it?
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alex Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204602#comment-17204602
]
Szehon Ho commented on HIVE-19253:
--
Hello guys, would it be possible to merge this patch? This broken API makes
it quite confusing.
Alex's approach seems reasonable good (accept tableType while keeping the old
way for compatibility). I took the liberty of rebasing and submitting a pull
request as per the new method, hope you guys don't mind.
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alex Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181770#comment-17181770 ] Szehon Ho commented on HIVE-24051: -- HI, [~chaosun] [~jxiang] would you guys help look at this small change? Thanks > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-24051: - Attachment: HIVE-24051.patch > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-24051: - Status: Patch Available (was: Open) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho reassigned HIVE-24051: Assignee: Szehon Ho > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-8950: Attachment: HIVE-8950.11.patch > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Ashish Singh >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.10.patch, > HIVE-8950.11.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, HIVE-8950.4.patch, > HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.7.patch, HIVE-8950.8.patch, > HIVE-8950.9.patch, HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22033) HiveServer2: fix delegation token renewal
[ https://issues.apache.org/jira/browse/HIVE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-22033: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master, thanks Ion for the contribution. > HiveServer2: fix delegation token renewal > - > > Key: HIVE-22033 > URL: https://issues.apache.org/jira/browse/HIVE-22033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.5 >Reporter: Ion Alberdi >Assignee: Ion Alberdi >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22033.2.patch, HIVE-22033.patch > > > Hello, the issue we faced (and a proposal for a fix) in our hive instances is > depicted at > [https://github.com/criteo-forks/hive/pull/24] > Reading the master branch of the project > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L147] > I think the same behavior is replicated there. > Long story short, *TokenStoreDelegationTokenSecretManager.renewToken*, does > not update the expiry date of a given token (as it does not get the updated > DelegationTokenInformation from *super.currentTokens*). > This makes any call to renewToken ineffective (the expiry date of the token > is not postponed). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22033) HiveServer2: fix delegation token renewal
[ https://issues.apache.org/jira/browse/HIVE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052167#comment-17052167 ] Szehon Ho commented on HIVE-22033: -- +1 on this patch. The test failure does not seem related (i ran manually and it passed) > HiveServer2: fix delegation token renewal > - > > Key: HIVE-22033 > URL: https://issues.apache.org/jira/browse/HIVE-22033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.5 >Reporter: Ion Alberdi >Assignee: Ion Alberdi >Priority: Major > Attachments: HIVE-22033.2.patch, HIVE-22033.patch > > > Hello, the issue we faced (and a proposal for a fix) in our hive instances is > depicted at > [https://github.com/criteo-forks/hive/pull/24] > Reading the master branch of the project > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L147] > I think the same behavior is replicated there. > Long story short, *TokenStoreDelegationTokenSecretManager.renewToken*, does > not update the expiry date of a given token (as it does not get the updated > DelegationTokenInformation from *super.currentTokens*). > This makes any call to renewToken ineffective (the expiry date of the token > is not postponed). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22033) HiveServer2: fix delegation token renewal
[ https://issues.apache.org/jira/browse/HIVE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-22033: - Attachment: HIVE-22033.2.patch > HiveServer2: fix delegation token renewal > - > > Key: HIVE-22033 > URL: https://issues.apache.org/jira/browse/HIVE-22033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.5 >Reporter: Ion Alberdi >Assignee: Ion Alberdi >Priority: Major > Attachments: HIVE-22033.2.patch, HIVE-22033.patch > > > Hello, the issue we faced (and a proposal for a fix) in our hive instances is > depicted at > [https://github.com/criteo-forks/hive/pull/24] > Reading the master branch of the project > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L147] > I think the same behavior is replicated there. > Long story short, *TokenStoreDelegationTokenSecretManager.renewToken*, does > not update the expiry date of a given token (as it does not get the updated > DelegationTokenInformation from *super.currentTokens*). > This makes any call to renewToken ineffective (the expiry date of the token > is not postponed). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22033) HiveServer2: fix delegation token renewal
[ https://issues.apache.org/jira/browse/HIVE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036265#comment-17036265 ] Szehon Ho commented on HIVE-22033: -- Rebasing on behalf of Jon. This is an important patch that we have had in prod for awhile at Criteo. It seems important to update the expiry date on the store-side, maybe in our environment as the token is accessed via different metastore instances. > HiveServer2: fix delegation token renewal > - > > Key: HIVE-22033 > URL: https://issues.apache.org/jira/browse/HIVE-22033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.5 >Reporter: Ion Alberdi >Assignee: Ion Alberdi >Priority: Major > Attachments: HIVE-22033.2.patch, HIVE-22033.patch > > > Hello, the issue we faced (and a proposal for a fix) in our hive instances is > depicted at > [https://github.com/criteo-forks/hive/pull/24] > Reading the master branch of the project > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L147] > I think the same behavior is replicated there. > Long story short, *TokenStoreDelegationTokenSecretManager.renewToken*, does > not update the expiry date of a given token (as it does not get the updated > DelegationTokenInformation from *super.currentTokens*). > This makes any call to renewToken ineffective (the expiry date of the token > is not postponed). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13457: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-13457.10.patch, HIVE-13457.11.patch, > HIVE-13457.12.patch, HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, > HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907408#comment-16907408 ] Szehon Ho commented on HIVE-13457: -- Test failures look not related. Committing original patch from Pawel with just checkstyle fixes. > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.10.patch, HIVE-13457.11.patch, > HIVE-13457.12.patch, HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, > HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907078#comment-16907078 ] Szehon Ho commented on HIVE-13457: -- More checkstyle fixes > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.10.patch, HIVE-13457.11.patch, > HIVE-13457.12.patch, HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, > HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13457: - Attachment: HIVE-13457.12.patch > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.10.patch, HIVE-13457.11.patch, > HIVE-13457.12.patch, HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, > HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906363#comment-16906363 ] Szehon Ho edited comment on HIVE-8950 at 8/13/19 4:19 PM: -- Made an amount of test fixes. Now that columns come from deserializer, the comment becomes "" instead of null. As part of trying to fix test, add an if check for setTableLocInTableProperties for non-external storage-handlers, and set ParquetHIveSerde.shouldStoreFieldsInMetastore to be false. The Serde flag seem to have been added after this patch was first made. was (Author: szehon): Made an amount of test fixes. Now that columns come from deserializer, the comment becomes "" instead of null. Also, add an if check for setTableLocInTableProperties for non-external storage-handlers, and set ParquetHIveSerde.shouldStoreFieldsInMetastore to be false. > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Ashish Singh >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.10.patch, > HIVE-8950.2.patch, HIVE-8950.3.patch, HIVE-8950.4.patch, HIVE-8950.5.patch, > HIVE-8950.6.patch, HIVE-8950.7.patch, HIVE-8950.8.patch, HIVE-8950.9.patch, > HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906363#comment-16906363 ] Szehon Ho commented on HIVE-8950: - Made an amount of test fixes. Now that columns come from deserializer, the comment becomes "" instead of null. Also, add an if check for setTableLocInTableProperties for non-external storage-handlers, and set ParquetHIveSerde.shouldStoreFieldsInMetastore to be false. > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Ashish Singh >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.10.patch, > HIVE-8950.2.patch, HIVE-8950.3.patch, HIVE-8950.4.patch, HIVE-8950.5.patch, > HIVE-8950.6.patch, HIVE-8950.7.patch, HIVE-8950.8.patch, HIVE-8950.9.patch, > HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-8950: Attachment: HIVE-8950.10.patch > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Ashish Singh >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.10.patch, > HIVE-8950.2.patch, HIVE-8950.3.patch, HIVE-8950.4.patch, HIVE-8950.5.patch, > HIVE-8950.6.patch, HIVE-8950.7.patch, HIVE-8950.8.patch, HIVE-8950.9.patch, > HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906358#comment-16906358 ] Szehon Ho commented on HIVE-13457: -- Missed a checkstyle fix > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.10.patch, HIVE-13457.11.patch, > HIVE-13457.3.patch, HIVE-13457.4.patch, HIVE-13457.5.patch, > HIVE-13457.6.patch, HIVE-13457.6.patch, HIVE-13457.7.patch, > HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13457: - Attachment: HIVE-13457.11.patch > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.10.patch, HIVE-13457.11.patch, > HIVE-13457.3.patch, HIVE-13457.4.patch, HIVE-13457.5.patch, > HIVE-13457.6.patch, HIVE-13457.6.patch, HIVE-13457.7.patch, > HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905953#comment-16905953 ] Szehon Ho commented on HIVE-13457: -- Fix imports corrupted during the rebase > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.10.patch, HIVE-13457.3.patch, > HIVE-13457.4.patch, HIVE-13457.5.patch, HIVE-13457.6.patch, > HIVE-13457.6.patch, HIVE-13457.7.patch, HIVE-13457.8.patch, > HIVE-13457.9.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13457: - Attachment: HIVE-13457.10.patch > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.10.patch, HIVE-13457.3.patch, > HIVE-13457.4.patch, HIVE-13457.5.patch, HIVE-13457.6.patch, > HIVE-13457.6.patch, HIVE-13457.7.patch, HIVE-13457.8.patch, > HIVE-13457.9.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905245#comment-16905245 ] Szehon Ho commented on HIVE-13457: -- Just fixing some simple checkstyle errors > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, > HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13457: - Attachment: HIVE-13457.9.patch > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.9.patch, HIVE-13457.patch, > HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905174#comment-16905174 ] Szehon Ho commented on HIVE-13457: -- Forgot to add new file .. > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13457: - Attachment: HIVE-13457.8.patch > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.8.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22033) HiveServer2: fix delegation token renewal
[ https://issues.apache.org/jira/browse/HIVE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-22033: - Assignee: Ion Alberdi Status: Patch Available (was: Open) > HiveServer2: fix delegation token renewal > - > > Key: HIVE-22033 > URL: https://issues.apache.org/jira/browse/HIVE-22033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.5 >Reporter: Ion Alberdi >Assignee: Ion Alberdi >Priority: Major > Attachments: HIVE-22033.patch > > > Hello, the issue we faced (and a proposal for a fix) in our hive instances is > depicted at > [https://github.com/criteo-forks/hive/pull/24] > Reading the master branch of the project > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L147] > I think the same behavior is replicated there. > Long story short, *TokenStoreDelegationTokenSecretManager.renewToken*, does > not update the expiry date of a given token (as it does not get the updated > DelegationTokenInformation from *super.currentTokens*). > This makes any call to renewToken ineffective (the expiry date of the token > is not postponed). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902913#comment-16902913 ] Szehon Ho commented on HIVE-13457: -- Also we have been using this patch for an internal Hive UI project in Criteo and it's been quite stable :) > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902911#comment-16902911 ] Szehon Ho commented on HIVE-13457: -- So someone recently has already added powermock to the parent pom, I think we are good. I +1 the patch and removed the redudant powermock version from service pom > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-13457) Create HS2 REST API endpoints for monitoring information
[ https://issues.apache.org/jira/browse/HIVE-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13457: - Attachment: HIVE-13457.7.patch > Create HS2 REST API endpoints for monitoring information > > > Key: HIVE-13457 > URL: https://issues.apache.org/jira/browse/HIVE-13457 > Project: Hive > Issue Type: Improvement >Reporter: Szehon Ho >Assignee: Pawel Szostek >Priority: Major > Attachments: HIVE-13457.3.patch, HIVE-13457.4.patch, > HIVE-13457.5.patch, HIVE-13457.6.patch, HIVE-13457.6.patch, > HIVE-13457.7.patch, HIVE-13457.patch, HIVE-13457.patch > > > Similar to what is exposed in HS2 webui in HIVE-12338, it would be nice if > other UI's like admin tools or Hue can access and display this information as > well. Hence, we will create some REST endpoints to expose this information. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho reassigned HIVE-8950: --- Assignee: Szehon Ho (was: Ashish Singh) > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, > HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.7.patch, > HIVE-8950.8.patch, HIVE-8950.9.patch, HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-8950: Assignee: Ashish Singh (was: Szehon Ho) Status: Patch Available (was: In Progress) > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Ashish Singh >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, > HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.7.patch, > HIVE-8950.8.patch, HIVE-8950.9.patch, HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-8950: Assignee: Ashish Singh (was: Gaurav Kumar) Status: In Progress (was: Patch Available) > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Ashish Singh >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, > HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.7.patch, > HIVE-8950.8.patch, HIVE-8950.9.patch, HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-8950: Attachment: HIVE-8950.9.patch > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Gaurav Kumar >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, > HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.7.patch, > HIVE-8950.8.patch, HIVE-8950.9.patch, HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902816#comment-16902816 ] Szehon Ho commented on HIVE-8950: - Hello, I'm a few years late :) but in Criteo we have been using this patch for a few months and it's really great. I'll +1 the patch and try to rebase and get it in, if its ok for everyone here > Add support in ParquetHiveSerde to create table schema from a parquet file > -- > > Key: HIVE-8950 > URL: https://issues.apache.org/jira/browse/HIVE-8950 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Singh >Assignee: Gaurav Kumar >Priority: Major > Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, > HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.7.patch, > HIVE-8950.8.patch, HIVE-8950.patch > > > PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without > having to specify the column names and types. As, parquet files store schema > in their footer, it is possible to generate hive schema from parquet file's > metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Affects Version/s: 2.0.0 3.0.0 Fix Version/s: 4.0.0 > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 3.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.5.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master, thanks a lot Aihua for the review! > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 3.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.5.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.5.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.5.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729655#comment-16729655 ] Szehon Ho commented on HIVE-21033: -- Fix checkstyle > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.4.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729603#comment-16729603 ] Szehon Ho commented on HIVE-21033: -- Patch creates new kind of PrintStream for SessionState, which I call 'SessionStream' to be able to handle not closing of System.out and System.err. As consequence, a lot of test classes need to change a little bit. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729601#comment-16729601 ] Szehon Ho commented on HIVE-21033: -- [~aihuaxu] Do you have time to look at this patch? > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: (was: HIVE-21033.2.patch) > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.2.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.2.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.3.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.2.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729097#comment-16729097 ] Szehon Ho commented on HIVE-21033: -- Rebasing > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Assignee: Szehon Ho Status: Patch Available (was: Open) > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > Attachments: HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: We had a custom client that did not handle closing the operations, until the end of the session. it is a mistake in the client, but it reveals kind of a vulnerability in HiveServer2 This happens if you have a session with (1) HiveCommandOperation and (2) SQLOperation and don't close them right after. For example a session that does the operations (set a=b; select * from foobar; ). When SQLOperation runs , it set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client closes the session, or disconnects which triggers closeSession() on the Thrift side. In this case, the closeSession closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. was: We had a custom client that did not handle closing the operations, until the end of the session. it is a mistake in the client, but it reveals kind of a vulnerability in HiveServer2 This happens if you have a session with (1) HiveCommandOperation and (2) SQLOperation and don't close them right after. For example a session that does the operations (set a=b; select * from foobar; ). When SQLOperation runs , it set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client closes the session, or disconnects. In this case, the Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This one closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: We had a custom client that did not handle closing the operations, until the end of the session. it is a mistake in the client, but it reveals kind of a vulnerability in HiveServer2 This happens if you have a session with (1) HiveCommandOperation and (2) SQLOperation and don't close them right after. For example a session that does the operations (set a=b; select * from foobar; ). When SQLOperation runs , it set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client closes the session, or disconnects which triggers closeSession() on the Thrift side. In this case, the closeSession closes all the operations, starting with HiveCommandOperation. This closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. was: We had a custom client that did not handle closing the operations, until the end of the session. it is a mistake in the client, but it reveals kind of a vulnerability in HiveServer2 This happens if you have a session with (1) HiveCommandOperation and (2) SQLOperation and don't close them right after. For example a session that does the operations (set a=b; select * from foobar; ). When SQLOperation runs , it set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client closes the session, or disconnects which triggers closeSession() on the Thrift side. In this case, the closeSession closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: We had a custom client that did not handle closing the operations, until the end of the session. it is a mistake in the client, but it reveals kind of a vulnerability in HiveServer2 This happens if you have a session with (1) HiveCommandOperation and (2) SQLOperation and don't close them right after. For example a session that does the operations (set a=b; select * from foobar; ). When SQLOperation runs , it set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client closes the session, or disconnects. In this case, the Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. was: We had a custom client that did not handle closing the operation or session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. This happens if you have a session with (1) HiveCommandOperation and (2) SQLOperation and don't close them right after. For example a session that does the operations (set a=b; select * from foobar; ). When SQLOperation runs , it set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client closes the session, or disconnects. In this case, the Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects. In this case, the > Session closes all the operations, starting with HiveCommandOperation. This > one closes all the streams, which is System.out and System.err as set by > SQLOperation earlier. Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: We had a custom client that did not handle closing the operation or session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. This happens if you have a session with (1) HiveCommandOperation and (2) SQLOperation and don't close them right after. For example a session that does the operations (set a=b; select * from foobar; ). When SQLOperation runs , it set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client closes the session, or disconnects. In this case, the Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. was: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the operation or session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. Basically you have a session with both HiveCommandOperation and SQLOperation. For example a session that does the operations (set a=b; select * from foobar; ). The SQLOperation runs last and set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: Ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > We had a custom client that did not handle closing the operation or session > on the error case. But it may also happen for any client that just > disconnects in the middle of this operation. > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects. In this case, the > Session closes all the operations, starting with HiveCommandOperation. This > one closes all the streams, which is System.out and System.err as set by > SQLOperation earlier. Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Summary: Forgetting to close operation cuts off any more HiveServer2 output (was: Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 output) > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > Its a bit tricky to reproduce, but we were able to do it (unfortunately) with > our custom client that did not handle closing the operation or session on the > error case. But it may also happen for any client that just disconnects in > the middle of this operation. > Basically you have a session with both HiveCommandOperation and SQLOperation. > For example a session that does the operations (set a=b; select * from > foobar; ). > The SQLOperation runs last and set SessionState.out and err to be System.out > and System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client terminates without closing the session. (In our case, a > SemanticException triggered it). The deleteContext is called, which closes > the session: Ref > [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] > The Session closes all the operations, starting with HiveCommandOperation. > This one closes all the streams, which is System.out and System.err as set by > SQLOperation earlier. Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the operation or session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. Basically you have a session with both HiveCommandOperation and SQLOperation. For example a session that does the operations (set a=b; select * from foobar; ). The SQLOperation runs last and set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: Ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. was: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. Basically you have a session with both HiveCommandOperation and SQLOperation. For example a session that does the operations (set a=b; select * from foobar; ). The SQLOperation runs last and set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: Ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. > Sudden disconnect for a session with set and SQL operation cuts off any more > HiveServer2 output > --- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > Its a bit tricky to reproduce, but we were able to do it (unfortunately) with > our custom client that did not handle closing the operation or session on the > error case. But it may also happen for any client that just disconnects in > the middle of this operation. > Basically you have a session with both HiveCommandOperation and SQLOperation. > For example a session that does the operations (set a=b; select * from > foobar; ). > The SQLOperation runs last and set SessionState.out and err to be System.out > and System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client terminates without closing the session. (In our case, a > SemanticException triggered it). The deleteContext is called, which closes > the session: Ref > [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] > The Session closes all the operations, starting with HiveCommandOperation. > This one closes all the streams, which is System.out and System.err as set by > SQLOperation earlier. Ref: > [HiveCommandOperation#
[jira] [Updated] (HIVE-21033) Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. Basically you have a session with both HiveCommandOperation and SQLOperation. For example a session that does the operations (set a=b; select * from foobar; ). The SQLOperation runs last and set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: Ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. was: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. Basically you have a session with both HiveCommandOperation and SQLOperation. For example a session that does the operations (set a=b; select * from foobar; ). The SQLOperation runs last and set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: Ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which it assumes is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. > Sudden disconnect for a session with set and SQL operation cuts off any more > HiveServer2 output > --- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > Its a bit tricky to reproduce, but we were able to do it (unfortunately) with > our custom client that did not handle closing the session on the error case. > But it may also happen for any client that just disconnects in the middle of > this operation. > Basically you have a session with both HiveCommandOperation and SQLOperation. > For example a session that does the operations (set a=b; select * from > foobar; ). > The SQLOperation runs last and set SessionState.out and err to be System.out > and System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client terminates without closing the session. (In our case, a > SemanticException triggered it). The deleteContext is called, which closes > the session: Ref > [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] > The Session closes all the operations, starting with HiveCommandOperation. > This one closes all the streams, which is System.out and System.err as set by > SQLOperation earlier. Ref: > [HiveCommandOperation#tearDownSession
[jira] [Updated] (HIVE-21033) Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 logs
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. But it may also happen for any client that just disconnects in the middle of this operation. Basically you have a session with both HiveCommandOperation and SQLOperation. For example a session that does the operations (set a=b; select * from foobar; ). The SQLOperation runs last and set SessionState.out and err to be System.out and System.err . Ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: Ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which it assumes is System.out and System.err as set by SQLOperation earlier. Ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 output appears as System.out and System.err are closed. was: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. Basically you have a session with both HiveCommandOperation and SQLOperation (set a=b; select * from foobar; ). Both will set up the session's out and err, with the SQLOperation setting it to be System.out and System.err . ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which it assumes is not System.err but was set so by SQLOperation. ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 logs appear. > Sudden disconnect for a session with set and SQL operation cuts off any more > HiveServer2 logs > - > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > Its a bit tricky to reproduce, but we were able to do it (unfortunately) with > our custom client that did not handle closing the session on the error case. > But it may also happen for any client that just disconnects in the middle of > this operation. > Basically you have a session with both HiveCommandOperation and SQLOperation. > For example a session that does the operations (set a=b; select * from > foobar; ). > The SQLOperation runs last and set SessionState.out and err to be System.out > and System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client terminates without closing the session. (In our case, a > SemanticException triggered it). The deleteContext is called, which closes > the session: Ref > [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] > The Session closes all the operations, starting with HiveCommandOperation. > This one closes all the streams, which it assumes is System.out and > System.err as set by SQLOperation earlier. Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101]
[jira] [Updated] (HIVE-21033) Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Summary: Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 output (was: Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 logs) > Sudden disconnect for a session with set and SQL operation cuts off any more > HiveServer2 output > --- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > Its a bit tricky to reproduce, but we were able to do it (unfortunately) with > our custom client that did not handle closing the session on the error case. > But it may also happen for any client that just disconnects in the middle of > this operation. > Basically you have a session with both HiveCommandOperation and SQLOperation. > For example a session that does the operations (set a=b; select * from > foobar; ). > The SQLOperation runs last and set SessionState.out and err to be System.out > and System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client terminates without closing the session. (In our case, a > SemanticException triggered it). The deleteContext is called, which closes > the session: Ref > [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] > The Session closes all the operations, starting with HiveCommandOperation. > This one closes all the streams, which it assumes is System.out and > System.err as set by SQLOperation earlier. Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 logs
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. Basically you have a session with both HiveCommandOperation and SQLOperation (set a=b; select * from foobar; ). Both will set up the session's out and err, with the SQLOperation setting it to be System.out and System.err . ref: [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] Then the client terminates without closing the session. (In our case, a SemanticException triggered it). The deleteContext is called, which closes the session: ref [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] The Session closes all the operations, starting with HiveCommandOperation. This one closes all the streams, which it assumes is not System.err but was set so by SQLOperation. ref: [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] After this, no more HiveServer2 logs appear. was: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. Basically you have a session with both HiveCommandOperation and SQLOperation (set a=b; select * from foobar;). Both will set up the session's out and err, with the SQLOperation setting it to be System.out and System.err ref: > Sudden disconnect for a session with set and SQL operation cuts off any more > HiveServer2 logs > - > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > Its a bit tricky to reproduce, but we were able to do it (unfortunately) with > our custom client that did not handle closing the session on the error case. > Basically you have a session with both HiveCommandOperation and SQLOperation > (set a=b; select * from foobar; ). > Both will set up the session's out and err, with the SQLOperation setting it > to be System.out and System.err . ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client terminates without closing the session. (In our case, a > SemanticException triggered it). The deleteContext is called, which closes > the session: ref > [ThriftBinaryCLIService#deleteContext|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java#L141] > The Session closes all the operations, starting with HiveCommandOperation. > This one closes all the streams, which it assumes is not System.err but was > set so by SQLOperation. ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 logs appear. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Sudden disconnect for a session with set and SQL operation cuts off any more HiveServer2 logs
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Description: Its a bit tricky to reproduce, but we were able to do it (unfortunately) with our custom client that did not handle closing the session on the error case. Basically you have a session with both HiveCommandOperation and SQLOperation (set a=b; select * from foobar;). Both will set up the session's out and err, with the SQLOperation setting it to be System.out and System.err ref: > Sudden disconnect for a session with set and SQL operation cuts off any more > HiveServer2 logs > - > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Priority: Major > > Its a bit tricky to reproduce, but we were able to do it (unfortunately) with > our custom client that did not handle closing the session on the error case. > Basically you have a session with both HiveCommandOperation and SQLOperation > (set a=b; select * from foobar;). Both will set up the session's out and > err, with the SQLOperation setting it to be System.out and System.err > ref: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master, thanks Vihang for the review
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Fix For: 4.0.0
>
> Attachments: HIVE-20786.2.patch, HIVE-20786.patch,
> hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler .
[jira] [Commented] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670912#comment-16670912
]
Szehon Ho commented on HIVE-20786:
--
So the problem was the top-level packaging was using an old version of maven
assembly before posix was even supported. This latest patch builds but also
upgrade the maven.assembly.plugin to the same one used in standalone-metastore.
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Attachment: (was: HIVE-20789.2.patch)
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO]
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Attachment: HIVE-20786.2.patch
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.2.patch, HIVE-20786.patch,
> hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIP
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Attachment: HIVE-20789.2.patch
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, HIVE-20789.2.patch,
> hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIP
