[jira] [Work logged] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?focusedWorklogId=651464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651464
]
ASF GitHub Bot logged work on HIVE-25522:
-
Author: ASF GitHub Bot
Created on: 16/Sep/21 05:26
Start Date: 16/Sep/21 05:26
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #2647:
URL: https://github.com/apache/hive/pull/2647#issuecomment-920589054
Thanks for the patch @szehon-ho!
Did not have time to check the code, but I can imagine us setting the
`connPool` to `null` just to signal that we have to reinitialize the whole
`TxnHandler` object. If this is the case we might end up with an invalid object
where some values are for the old conf, some values are for the new.
I will check the code if I have some time later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 651464)
Time Spent: 20m (was: 10m)
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
>
[jira] [Work logged] (HIVE-24762) StringValueBoundaryScanner ignores boundary which leads to incorrect results
[
https://issues.apache.org/jira/browse/HIVE-24762?focusedWorklogId=651411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651411
]
ASF GitHub Bot logged work on HIVE-24762:
-
Author: ASF GitHub Bot
Created on: 16/Sep/21 00:10
Start Date: 16/Sep/21 00:10
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #1965:
URL: https://github.com/apache/hive/pull/1965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 651411)
Time Spent: 2h 10m (was: 2h)
> StringValueBoundaryScanner ignores boundary which leads to incorrect results
> -
>
> Key: HIVE-24762
> URL: https://issues.apache.org/jira/browse/HIVE-24762
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/ValueBoundaryScanner.java#L901
> {code}
> public boolean isDistanceGreater(Object v1, Object v2, int amt) {
> ...
> return s1 != null && s2 != null && s1.compareTo(s2) > 0;
> {code}
> Like other boundary scanners, StringValueBoundaryScanner should take amt into
> account, otherwise it'll result in the same range regardless of the given
> window size. This typically affects queries where the range is defined on a
> string column:
> {code}
> select p_mfgr, p_name, p_retailprice,
> count(*) over(partition by p_mfgr order by p_name range between 1 preceding
> and current row) as cs1,
> count(*) over(partition by p_mfgr order by p_name range between 3 preceding
> and current row) as cs2
> from vector_ptf_part_simple_orc;
> {code}
> with "> 0" cs1 and cs2 will be calculated on the same window, so cs1 == cs2,
> but actually it should be different, this is the correct result (see "almond
> antique olive coral navajo"):
> {code}
> +-+-+--+--+
> | p_mfgr | p_name| cs1 | cs2
> |
> +-+-+--+--+
> | Manufacturer#1 | almond antique burnished rose metallic | 2| 2
> |
> | Manufacturer#1 | almond antique burnished rose metallic | 2| 2
> |
> | Manufacturer#1 | almond antique chartreuse lavender yellow | 6| 6
> |
> | Manufacturer#1 | almond antique chartreuse lavender yellow | 6| 6
> |
> | Manufacturer#1 | almond antique chartreuse lavender yellow | 6| 6
> |
> | Manufacturer#1 | almond antique chartreuse lavender yellow | 6| 6
> |
> | Manufacturer#1 | almond antique salmon chartreuse burlywood | 1| 1
> |
> | Manufacturer#1 | almond aquamarine burnished black steel | 1| 8
> |
> | Manufacturer#1 | almond aquamarine pink moccasin thistle | 4| 4
> |
> | Manufacturer#1 | almond aquamarine pink moccasin thistle | 4| 4
> |
> | Manufacturer#1 | almond aquamarine pink moccasin thistle | 4| 4
> |
> | Manufacturer#1 | almond aquamarine pink moccasin thistle | 4| 4
> |
> | Manufacturer#2 | almond antique violet chocolate turquoise | 1| 1
> |
> | Manufacturer#2 | almond antique violet turquoise frosted | 3| 3
> |
> | Manufacturer#2 | almond antique violet turquoise frosted | 3| 3
> |
> | Manufacturer#2 | almond antique violet turquoise frosted | 3| 3
> |
> | Manufacturer#2 | almond aquamarine midnight light salmon | 1| 5
> |
> | Manufacturer#2 | almond aquamarine rose maroon antique | 2| 2
> |
> | Manufacturer#2 | almond aquamarine rose maroon antique | 2| 2
> |
> | Manufacturer#2 | almond aquamarine sandy cyan gainsboro | 3| 3
> |
> | Manufacturer#3 | almond antique chartreuse khaki white | 1| 1
> |
> | Manufacturer#3 | almond antique forest lavender goldenrod| 4| 5
> |
> | Manufacturer#3 | almond antique forest lavender goldenrod| 4| 5
> |
> | Manufacturer#3 | almond antique forest lavender goldenrod| 4| 5
> |
> | Manufacturer#3 | almond antique forest lavender goldenrod| 4| 5
> |
> | Manufacturer#3 | almond antique metallic orange dim | 1| 1
> |
> |
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25522:
--
Labels: pull-request-available (was: )
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
>
[jira] [Work logged] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?focusedWorklogId=651376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651376
]
ASF GitHub Bot logged work on HIVE-25522:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 22:23
Start Date: 15/Sep/21 22:23
Worklog Time Spent: 10m
Work Description: szehon-ho opened a new pull request #2647:
URL: https://github.com/apache/hive/pull/2647
### What changes were proposed in this pull request?
* This fixes https://issues.apache.org/jira/browse/HIVE-25522
* There are two options, either make the initialization static and kill HMS
if there is an error, or keep it lazy. Went with second approach as there seem
to be db connections that are taken and don't need to be if nobody uses any
txnHandler methods.
* Make the initailization setConf method be idempotent by checking each of
the static variables it sets and able to resume if a particular variable is not
set.
* Also some refactor to push down verbose catch blocks as much as possible.
### Why are the changes needed?
* See https://issues.apache.org/jira/browse/HIVE-25522
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Running unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 651376)
Remaining Estimate: 0h
Time Spent: 10m
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
>
[jira] [Updated] (HIVE-25528) Avoid recalculating types after CBO on second AST pass
[ https://issues.apache.org/jira/browse/HIVE-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25528: -- Labels: pull-request-available (was: ) > Avoid recalculating types after CBO on second AST pass > -- > > Key: HIVE-25528 > URL: https://issues.apache.org/jira/browse/HIVE-25528 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > It should be possible to avoid recalculating and reevaluating types on the > second pass after going through CBO. CBO is making the effort to change the > types so to reassess them is a waste of time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25528) Avoid recalculating types after CBO on second AST pass
[ https://issues.apache.org/jira/browse/HIVE-25528?focusedWorklogId=651317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651317 ] ASF GitHub Bot logged work on HIVE-25528: - Author: ASF GitHub Bot Created on: 15/Sep/21 20:27 Start Date: 15/Sep/21 20:27 Worklog Time Spent: 10m Work Description: scarlin-cloudera opened a new pull request #2646: URL: https://github.com/apache/hive/pull/2646 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 651317) Remaining Estimate: 0h Time Spent: 10m > Avoid recalculating types after CBO on second AST pass > -- > > Key: HIVE-25528 > URL: https://issues.apache.org/jira/browse/HIVE-25528 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It should be possible to avoid recalculating and reevaluating types on the > second pass after going through CBO. CBO is making the effort to change the > types so to reassess them is a waste of time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25528) Avoid recalculating types after CBO on second AST pass
[ https://issues.apache.org/jira/browse/HIVE-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Carlin reassigned HIVE-25528: --- Assignee: Steve Carlin > Avoid recalculating types after CBO on second AST pass > -- > > Key: HIVE-25528 > URL: https://issues.apache.org/jira/browse/HIVE-25528 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > > It should be possible to avoid recalculating and reevaluating types on the > second pass after going through CBO. CBO is making the effort to change the > types so to reassess them is a waste of time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25518) CompactionTxHandler NPE if no CompactionInfo
[
https://issues.apache.org/jira/browse/HIVE-25518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csomor Viktor updated HIVE-25518:
-
Attachment: HIVE_25518_CompactionTxnHandle_NPE.txt
> CompactionTxHandler NPE if no CompactionInfo
>
>
> Key: HIVE-25518
> URL: https://issues.apache.org/jira/browse/HIVE-25518
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Csomor Viktor
>Assignee: Csomor Viktor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE_25518_CompactionTxnHandle_NPE.txt
>
>
> If no {{CompactionInfo}} is provided to the
> {{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
> the method. No information inside the COMPLETED_COMPACTION info.
> Stacktrace:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler -
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:1116)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.mark_failed(HMSHandler.java:8716)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy13.mark_failed(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23846)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23825)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25518) CompactionTxHandler NPE if no CompactionInfo
[
https://issues.apache.org/jira/browse/HIVE-25518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415691#comment-17415691
]
Csomor Viktor commented on HIVE-25518:
--
[~zabetak] thanks for letting me know! I've attached the stacktrace.
(/) [^HIVE_25518_CompactionTxnHandle_NPE.txt]
> CompactionTxHandler NPE if no CompactionInfo
>
>
> Key: HIVE-25518
> URL: https://issues.apache.org/jira/browse/HIVE-25518
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Csomor Viktor
>Assignee: Csomor Viktor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE_25518_CompactionTxnHandle_NPE.txt
>
>
> If no {{CompactionInfo}} is provided to the
> {{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
> the method. No information inside the COMPLETED_COMPACTION info.
> Stacktrace:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler -
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:1116)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.mark_failed(HMSHandler.java:8716)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy13.mark_failed(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23846)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23825)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25518) CompactionTxHandler NPE if no CompactionInfo
[
https://issues.apache.org/jira/browse/HIVE-25518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csomor Viktor updated HIVE-25518:
-
Description:
If no {{CompactionInfo}} is provided to the
{{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
the method. No information inside the COMPLETED_COMPACTION info.
Stacktrace:
{noformat}
20:47:44.916 [TThreadPoolServer WorkerProcess-%d] ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler -
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:1116)
at
org.apache.hadoop.hive.metastore.HMSHandler.mark_failed(HMSHandler.java:8716)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy13.mark_failed(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23846)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23825)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:If no {{CompactionInfo}} is provided to the
{{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
the method. No information inside the COMPLETED_COMPACTION info.
> CompactionTxHandler NPE if no CompactionInfo
>
>
> Key: HIVE-25518
> URL: https://issues.apache.org/jira/browse/HIVE-25518
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Csomor Viktor
>Assignee: Csomor Viktor
>Priority: Major
> Fix For: 4.0.0
>
>
> If no {{CompactionInfo}} is provided to the
> {{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
> the method. No information inside the COMPLETED_COMPACTION info.
> Stacktrace:
> {noformat}
> 20:47:44.916 [TThreadPoolServer WorkerProcess-%d] ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler -
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:1116)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.mark_failed(HMSHandler.java:8716)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy13.mark_failed(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23846)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23825)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native
[jira] [Updated] (HIVE-25518) CompactionTxHandler NPE if no CompactionInfo
[
https://issues.apache.org/jira/browse/HIVE-25518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csomor Viktor updated HIVE-25518:
-
Description:
If no {{CompactionInfo}} is provided to the
{{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
the method. No information inside the COMPLETED_COMPACTION info.
Stacktrace:
{noformat}
[TThreadPoolServer WorkerProcess-%d] ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler -
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:1116)
at
org.apache.hadoop.hive.metastore.HMSHandler.mark_failed(HMSHandler.java:8716)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy13.mark_failed(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23846)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23825)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
If no {{CompactionInfo}} is provided to the
{{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
the method. No information inside the COMPLETED_COMPACTION info.
Stacktrace:
{noformat}
20:47:44.916 [TThreadPoolServer WorkerProcess-%d] ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler -
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:1116)
at
org.apache.hadoop.hive.metastore.HMSHandler.mark_failed(HMSHandler.java:8716)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy13.mark_failed(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23846)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$mark_failed.getResult(ThriftHiveMetastore.java:23825)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> CompactionTxHandler NPE if no CompactionInfo
>
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415278#comment-17415278
]
Szehon Ho edited comment on HIVE-25522 at 9/15/21, 4:42 PM:
[~csun] helped me to reason through the code and pointed out the potential
error, thanks!
It looks like the TxnHandler's initialization block sets a lot of static
variables llke connPool, sqlGenerator, etc.. But it skips it if only connPool
is set. So it's possible that connPool was successfully set but the thread
crashed and did not set the other variables. Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L372
It looks like TxnHandler's are thread local and are constructed on demand, so a
fatal error in initialization does not kill the HMS server.
was (Author: szehon):
[~csun] helped me to reason through the code and pointed out the potential
error, thanks!
It looks like the TxnHandler's initialization block sets a lot of static
variables llke connPool, sqlGenerator, etc.. But it skips it if only connPool
is set. So it's possible that connPool was successfully set but the thread
crashed and did not set the other variables. Ref:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L372
It looks like TxnHandler's are thread local and are constructed on demand, so a
fatal error in one thread does not kill the HMS server.
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
>
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415625#comment-17415625
]
Szehon Ho edited comment on HIVE-25522 at 9/15/21, 4:41 PM:
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after connPool is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
was (Author: szehon):
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after connPool
is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415625#comment-17415625
]
Szehon Ho edited comment on HIVE-25522 at 9/15/21, 4:41 PM:
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after connPool
is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
was (Author: szehon):
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after dbConn is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
>
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Affects Version/s: 4.0.0
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
> at
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415625#comment-17415625
]
Szehon Ho commented on HIVE-25522:
--
Just some context how this issue happened for us, in our case the thread looks
like it crashed trying to get more db connections to calculate the other
variables after dbConn is assigned around
https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L313,
as database was low on connections at the time
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415624#comment-17415624
]
Szehon Ho commented on HIVE-25522:
--
Hi [~pvary] thanks for the reply, I found the issue last night, can give a stab
at a pr today. Thanks
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
[jira] [Assigned] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-25522:
Assignee: Szehon Ho
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
> at
[jira] [Updated] (HIVE-25527) LLAP Scheduler task exits with fatal error if the executor node is down.
[ https://issues.apache.org/jira/browse/HIVE-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-25527: -- Parent: HIVE-24913 Issue Type: Sub-task (was: Bug) > LLAP Scheduler task exits with fatal error if the executor node is down. > > > Key: HIVE-25527 > URL: https://issues.apache.org/jira/browse/HIVE-25527 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In case the executor host has gone down, activeInstances will be updated with > null. So we need to check for empty/null values before accessing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25527) LLAP Scheduler task exits with fatal error if the executor node is down.
[
https://issues.apache.org/jira/browse/HIVE-25527?focusedWorklogId=651135=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651135
]
ASF GitHub Bot logged work on HIVE-25527:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 14:53
Start Date: 15/Sep/21 14:53
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #2645:
URL: https://github.com/apache/hive/pull/2645#discussion_r709269471
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -1447,8 +1447,14 @@ private SelectHostResult selectHost(TaskInfo request,
Map
if (request.shouldForceLocality()) {
requestedHostsWillBecomeAvailable = true;
} else {
- LlapServiceInstance inst =
activeInstances.getByHost(host).stream().findFirst().get();
- NodeInfo nodeInfo =
instanceToNodeMap.get(inst.getWorkerIdentity());
+ Set instanceTypes =
activeInstances.getByHost(host);
Review comment:
Looks like this may happen when a node goes down between
getResourceAvailability() call until selectHost() is triggered. Following the
previous logic I believe the check should be performed at the same level as:
```availableHostMap.containsKey(host)``` as these type of requests should
not be checking for requestedHostsWillBecomeAvailable. Does it make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 651135)
Time Spent: 0.5h (was: 20m)
> LLAP Scheduler task exits with fatal error if the executor node is down.
>
>
> Key: HIVE-25527
> URL: https://issues.apache.org/jira/browse/HIVE-25527
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In case the executor host has gone down, activeInstances will be updated with
> null. So we need to check for empty/null values before accessing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25527) LLAP Scheduler task exits with fatal error if the executor node is down.
[
https://issues.apache.org/jira/browse/HIVE-25527?focusedWorklogId=651134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651134
]
ASF GitHub Bot logged work on HIVE-25527:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 14:52
Start Date: 15/Sep/21 14:52
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #2645:
URL: https://github.com/apache/hive/pull/2645#discussion_r709269471
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -1447,8 +1447,14 @@ private SelectHostResult selectHost(TaskInfo request,
Map
if (request.shouldForceLocality()) {
requestedHostsWillBecomeAvailable = true;
} else {
- LlapServiceInstance inst =
activeInstances.getByHost(host).stream().findFirst().get();
- NodeInfo nodeInfo =
instanceToNodeMap.get(inst.getWorkerIdentity());
+ Set instanceTypes =
activeInstances.getByHost(host);
Review comment:
Looks like this may happen when a node goes down between
getResourceAvailability() call until selectHost() is triggered. Following the
previous logic I believe the check should be performed at the same level as:
```availableHostMap.containsKey(host)``` as these type of requests should
not be waiting for requestedHostsWillBecomeAvailable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 651134)
Time Spent: 20m (was: 10m)
> LLAP Scheduler task exits with fatal error if the executor node is down.
>
>
> Key: HIVE-25527
> URL: https://issues.apache.org/jira/browse/HIVE-25527
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In case the executor host has gone down, activeInstances will be updated with
> null. So we need to check for empty/null values before accessing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25527) LLAP Scheduler task exits with fatal error if the executor node is down.
[ https://issues.apache.org/jira/browse/HIVE-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25527: -- Labels: pull-request-available (was: ) > LLAP Scheduler task exits with fatal error if the executor node is down. > > > Key: HIVE-25527 > URL: https://issues.apache.org/jira/browse/HIVE-25527 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In case the executor host has gone down, activeInstances will be updated with > null. So we need to check for empty/null values before accessing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25527) LLAP Scheduler task exits with fatal error if the executor node is down.
[ https://issues.apache.org/jira/browse/HIVE-25527?focusedWorklogId=651113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651113 ] ASF GitHub Bot logged work on HIVE-25527: - Author: ASF GitHub Bot Created on: 15/Sep/21 14:18 Start Date: 15/Sep/21 14:18 Worklog Time Spent: 10m Work Description: maheshk114 opened a new pull request #2645: URL: https://github.com/apache/hive/pull/2645 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 651113) Remaining Estimate: 0h Time Spent: 10m > LLAP Scheduler task exits with fatal error if the executor node is down. > > > Key: HIVE-25527 > URL: https://issues.apache.org/jira/browse/HIVE-25527 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In case the executor host has gone down, activeInstances will be updated with > null. So we need to check for empty/null values before accessing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25527) LLAP Scheduler task exits with fatal error if the executor node is down.
[ https://issues.apache.org/jira/browse/HIVE-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-25527: -- > LLAP Scheduler task exits with fatal error if the executor node is down. > > > Key: HIVE-25527 > URL: https://issues.apache.org/jira/browse/HIVE-25527 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > > In case the executor host has gone down, activeInstances will be updated with > null. So we need to check for empty/null values before accessing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25509) CLIService.closeOperation should not fail if operation handle is not present
[ https://issues.apache.org/jira/browse/HIVE-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-25509: Description: I think, if we want the close operation to be safe and idempotent, we should not fail on client side if 2 different close requests come in (due to socket timeout + retry) for the same operation handle (and second fails because operation handle was already removed for the first request which then socket timed out) > CLIService.closeOperation should not fail if operation handle is not present > > > Key: HIVE-25509 > URL: https://issues.apache.org/jira/browse/HIVE-25509 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > I think, if we want the close operation to be safe and idempotent, we should > not fail on client side if 2 different close requests come in (due to socket > timeout + retry) for the same operation handle (and second fails because > operation handle was already removed for the first request which then socket > timed out) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25485) Transform selects of literals under a UNION ALL to inline table scan
[
https://issues.apache.org/jira/browse/HIVE-25485?focusedWorklogId=651058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651058
]
ASF GitHub Bot logged work on HIVE-25485:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 12:27
Start Date: 15/Sep/21 12:27
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #2608:
URL: https://github.com/apache/hive/pull/2608#discussion_r709113570
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveTransformSimpleSelectsToInlineTableInUnion.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException;
+import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.TraitsUtil;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableFunctionScan;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveUnion;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.translator.SqlFunctionConverter;
+import org.apache.hadoop.hive.ql.parse.SemanticAnalyzer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.ImmutableList;
+
+/**
+ * Transforms SELECTS of literals under UNION ALL into inline table scans.
+ */
Review comment:
Could you please add a simple example?
Also a input and output plan something like
from
```
HiveUnion(all=true)
HiveProject(a=[1])
...
HiveProject(a=[42])
...
```
to
```
HiveProject(a=[$0])
HiveTableScan(table=...)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 651058)
Time Spent: 40m (was: 0.5h)
> Transform selects of literals under a UNION ALL to inline table scan
>
>
> Key: HIVE-25485
> URL: https://issues.apache.org/jira/browse/HIVE-25485
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code}
> select 1
> union all
> select 1
> union all
> [...]
> union all
> select 1
> {code}
> results in a very big plan; which will have vertexes proportional to the
> number of union all branch - hence it could be slow to execute it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25485) Transform selects of literals under a UNION ALL to inline table scan
[
https://issues.apache.org/jira/browse/HIVE-25485?focusedWorklogId=651057=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651057
]
ASF GitHub Bot logged work on HIVE-25485:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 12:26
Start Date: 15/Sep/21 12:26
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #2608:
URL: https://github.com/apache/hive/pull/2608#discussion_r709110069
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveTransformSimpleSelectsToInlineTableInUnion.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException;
+import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.TraitsUtil;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableFunctionScan;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveUnion;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.translator.SqlFunctionConverter;
+import org.apache.hadoop.hive.ql.parse.SemanticAnalyzer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.ImmutableList;
+
+/**
+ * Transforms SELECTS of literals under UNION ALL into inline table scans.
+ */
+public class HiveTransformSimpleSelectsToInlineTableInUnion extends RelOptRule
{
Review comment:
It is a long name but in case of other rules we add the word `Rule`.
How about `HiveUnionSimpleSelectsToInlineTableRule` ?
##
File path: ql/src/test/results/clientpositive/llap/union_literals.q.out
##
@@ -0,0 +1,397 @@
+PREHOOK: query: explain
+SELECT * FROM (
+ VALUES(1, '1'),
+ (2, 'orange'),
+ (5, 'yellow'),
+ (10, 'green'),
+ (11, 'blue'),
+ (12, 'indigo'),
+ (20, 'violet'))
+ AS Colors
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here
+POSTHOOK: query: explain
+SELECT * FROM (
+ VALUES(1, '1'),
+ (2, 'orange'),
+ (5, 'yellow'),
+ (10, 'green'),
+ (11, 'blue'),
+ (12, 'indigo'),
+ (20, 'violet'))
+ AS Colors
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here
+STAGE DEPENDENCIES:
+ Stage-0 is a root stage
+
+STAGE PLANS:
+ Stage: Stage-0
+Fetch Operator
+ limit: -1
+ Processor Tree:
+TableScan
+ alias: _dummy_table
+ Row Limit Per Split: 1
+ Select Operator
+expressions: array(const struct(1,'1'),const
struct(2,'orange'),const struct(5,'yellow'),const struct(10,'green'),const
struct(11,'blue'),const struct(12,'indigo'),const struct(20,'violet')) (type:
array>)
+outputColumnNames: _col0
+UDTF Operator
+ function name: inline
+ Select Operator
+expressions: col1 (type: int), col2 (type: string)
+outputColumnNames: _col0, _col1
+ListSink
+
+PREHOOK: query: explain
+SELECT * FROM (
+ VALUES(1, '1'),
+ (2, 'orange'),
+ (5, 'yellow'),
+
[jira] [Work logged] (HIVE-25485) Transform selects of literals under a UNION ALL to inline table scan
[
https://issues.apache.org/jira/browse/HIVE-25485?focusedWorklogId=651033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-651033
]
ASF GitHub Bot logged work on HIVE-25485:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 11:34
Start Date: 15/Sep/21 11:34
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #2608:
URL: https://github.com/apache/hive/pull/2608#issuecomment-919939445
@kasakrisz could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 651033)
Time Spent: 20m (was: 10m)
> Transform selects of literals under a UNION ALL to inline table scan
>
>
> Key: HIVE-25485
> URL: https://issues.apache.org/jira/browse/HIVE-25485
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code}
> select 1
> union all
> select 1
> union all
> [...]
> union all
> select 1
> {code}
> results in a very big plan; which will have vertexes proportional to the
> number of union all branch - hence it could be slow to execute it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415440#comment-17415440
]
Peter Vary commented on HIVE-25522:
---
[~szehon]: Seems like a legitimate problem even with the master branch. Do you
plan to raise a pull request with a fix?
CC: [~dkuzmenko]
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Priority: Major
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
>
[jira] [Commented] (HIVE-25518) CompactionTxHandler NPE if no CompactionInfo
[
https://issues.apache.org/jira/browse/HIVE-25518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415421#comment-17415421
]
Stamatis Zampetakis commented on HIVE-25518:
[~vcsomor] It's always helpful to include stacktraces (inside \{noformat\} tag)
in case of exceptions in the JIRA since users can easily find it in the future
by searching Google.
> CompactionTxHandler NPE if no CompactionInfo
>
>
> Key: HIVE-25518
> URL: https://issues.apache.org/jira/browse/HIVE-25518
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Csomor Viktor
>Assignee: Csomor Viktor
>Priority: Major
> Fix For: 4.0.0
>
>
> If no {{CompactionInfo}} is provided to the
> {{CompactionTxHandler#markFailed()}} then an NPE happens at the beginning of
> the method. No information inside the COMPLETED_COMPACTION info.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25521) Data corruption when concatenating files with different compressions in same table/partition
[ https://issues.apache.org/jira/browse/HIVE-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-25521: --- Summary: Data corruption when concatenating files with different compressions in same table/partition (was: Fix concatenate file handling when files of different compressions are in same table/partition.) > Data corruption when concatenating files with different compressions in same > table/partition > > > Key: HIVE-25521 > URL: https://issues.apache.org/jira/browse/HIVE-25521 > Project: Hive > Issue Type: Bug >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > > Currently if files of different compressions are in same directory then > concatenate can fail and cause data corruption. This happens because file can > be moved by one task as incompatible file and the other tasks will fail after > this. > > This issue is addressed in this Jira by only processing a file in one task > where offset 0 is process and ignoring the the file in all other tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25524) Mergefiletask can cause some lack of package problems
[
https://issues.apache.org/jira/browse/HIVE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415414#comment-17415414
]
Stamatis Zampetakis commented on HIVE-25524:
[~luguangming] It's better to use text inside \{noformat\} tags instead of
images for including stacktraces. Images are not searchable.
> Mergefiletask can cause some lack of package problems
> -
>
> Key: HIVE-25524
> URL: https://issues.apache.org/jira/browse/HIVE-25524
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: GuangMing Lu
>Priority: Major
> Attachments: image-2021-09-15-11-32-24-635.png,
> image-2021-09-15-11-33-24-743.png
>
>
> Mergefiletask does not load hive.aux.aux.ars.path's bag, which can cause some
> lack of package problems, such as follow test sql:
> create table tb_a (col5 date, col6 timestamp) stored as textfile;
> insert into tb_a select '2021-08-23','2021-08-23 09:30:00';
> set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
> create table tb_hive_orc stored as orc as select * from tb_a;
> !image-2021-09-15-11-32-24-635.png!
> !image-2021-09-15-11-33-24-743.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25414) Optimise Hive::addWriteNotificationLog: Reduce FS call per notification
[ https://issues.apache.org/jira/browse/HIVE-25414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-25414: -- Assignee: mahesh kumar behera > Optimise Hive::addWriteNotificationLog: Reduce FS call per notification > --- > > Key: HIVE-25414 > URL: https://issues.apache.org/jira/browse/HIVE-25414 > Project: Hive > Issue Type: Sub-task >Reporter: Rajesh Balamohan >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > AddWriteNotification is slow due to FS interactions (i.e to get the set of > insert file information). This can be avoided as FileStatus can be passed > instead of Path from parent methods. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3572] > > [https://github.com/apache/hive/blob/96b39cd5190f0cfadb677e3f3b7ead1d663921b2/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3620] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25438) Update partition column stats fails with invalid syntax error for MySql
[ https://issues.apache.org/jira/browse/HIVE-25438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera resolved HIVE-25438. Resolution: Fixed > Update partition column stats fails with invalid syntax error for MySql > --- > > Key: HIVE-25438 > URL: https://issues.apache.org/jira/browse/HIVE-25438 > Project: Hive > Issue Type: Sub-task >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The quotes are not supported by mysql if ANSI_QUOTES is not set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25414) Optimise Hive::addWriteNotificationLog: Reduce FS call per notification
[ https://issues.apache.org/jira/browse/HIVE-25414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera resolved HIVE-25414. Resolution: Fixed > Optimise Hive::addWriteNotificationLog: Reduce FS call per notification > --- > > Key: HIVE-25414 > URL: https://issues.apache.org/jira/browse/HIVE-25414 > Project: Hive > Issue Type: Sub-task >Reporter: Rajesh Balamohan >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > AddWriteNotification is slow due to FS interactions (i.e to get the set of > insert file information). This can be avoided as FileStatus can be passed > instead of Path from parent methods. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3572] > > [https://github.com/apache/hive/blob/96b39cd5190f0cfadb677e3f3b7ead1d663921b2/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3620] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25526) Run create_table Q test from TestCliDriver
[ https://issues.apache.org/jira/browse/HIVE-25526?focusedWorklogId=650968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650968 ] ASF GitHub Bot logged work on HIVE-25526: - Author: ASF GitHub Bot Created on: 15/Sep/21 08:43 Start Date: 15/Sep/21 08:43 Worklog Time Spent: 10m Work Description: veghlaci05 commented on pull request #2643: URL: https://github.com/apache/hive/pull/2643#issuecomment-919821411 @lcspinter Please review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650968) Time Spent: 20m (was: 10m) > Run create_table Q test from TestCliDriver > -- > > Key: HIVE-25526 > URL: https://issues.apache.org/jira/browse/HIVE-25526 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Laszlo Vegh >Assignee: Laszlo Vegh >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > create_table QTest should be picked up by the TestCliDriver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25526) Run create_table Q test from TestCliDriver
[ https://issues.apache.org/jira/browse/HIVE-25526?focusedWorklogId=650966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650966 ] ASF GitHub Bot logged work on HIVE-25526: - Author: ASF GitHub Bot Created on: 15/Sep/21 08:42 Start Date: 15/Sep/21 08:42 Worklog Time Spent: 10m Work Description: veghlaci05 opened a new pull request #2643: URL: https://github.com/apache/hive/pull/2643 ### What changes were proposed in this pull request? Introduce new test in the TestCliDriver. ### Why are the changes needed? Improve test coverage. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? QTest -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650966) Remaining Estimate: 0h Time Spent: 10m > Run create_table Q test from TestCliDriver > -- > > Key: HIVE-25526 > URL: https://issues.apache.org/jira/browse/HIVE-25526 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Laszlo Vegh >Assignee: Laszlo Vegh >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > create_table QTest should be picked up by the TestCliDriver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25526) Run create_table Q test from TestCliDriver
[ https://issues.apache.org/jira/browse/HIVE-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25526: -- Labels: pull-request-available (was: ) > Run create_table Q test from TestCliDriver > -- > > Key: HIVE-25526 > URL: https://issues.apache.org/jira/browse/HIVE-25526 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Laszlo Vegh >Assignee: Laszlo Vegh >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > create_table QTest should be picked up by the TestCliDriver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25526) Run create_table Q test from TestCliDriver
[ https://issues.apache.org/jira/browse/HIVE-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Vegh reassigned HIVE-25526: -- > Run create_table Q test from TestCliDriver > -- > > Key: HIVE-25526 > URL: https://issues.apache.org/jira/browse/HIVE-25526 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Laszlo Vegh >Assignee: Laszlo Vegh >Priority: Minor > > create_table QTest should be picked up by the TestCliDriver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25317) Relocate dependencies in shaded hive-exec module
[ https://issues.apache.org/jira/browse/HIVE-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved HIVE-25317. Resolution: Won't Fix > Relocate dependencies in shaded hive-exec module > > > Key: HIVE-25317 > URL: https://issues.apache.org/jira/browse/HIVE-25317 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.8 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > When we want to use shaded version of hive-exec (i.e., w/o classifier), more > dependencies conflict with Spark. We need to relocate these dependencies too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25317) Relocate dependencies in shaded hive-exec module
[ https://issues.apache.org/jira/browse/HIVE-25317?focusedWorklogId=650952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650952 ] ASF GitHub Bot logged work on HIVE-25317: - Author: ASF GitHub Bot Created on: 15/Sep/21 07:31 Start Date: 15/Sep/21 07:31 Worklog Time Spent: 10m Work Description: viirya closed pull request #2459: URL: https://github.com/apache/hive/pull/2459 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650952) Time Spent: 3h (was: 2h 50m) > Relocate dependencies in shaded hive-exec module > > > Key: HIVE-25317 > URL: https://issues.apache.org/jira/browse/HIVE-25317 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.8 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > When we want to use shaded version of hive-exec (i.e., w/o classifier), more > dependencies conflict with Spark. We need to relocate these dependencies too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25317) Relocate dependencies in shaded hive-exec module
[ https://issues.apache.org/jira/browse/HIVE-25317?focusedWorklogId=650951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650951 ] ASF GitHub Bot logged work on HIVE-25317: - Author: ASF GitHub Bot Created on: 15/Sep/21 07:31 Start Date: 15/Sep/21 07:31 Worklog Time Spent: 10m Work Description: viirya commented on pull request #2459: URL: https://github.com/apache/hive/pull/2459#issuecomment-919772056 Close this as we are taking a different direction to shade dependencies of Hive at Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650951) Time Spent: 2h 50m (was: 2h 40m) > Relocate dependencies in shaded hive-exec module > > > Key: HIVE-25317 > URL: https://issues.apache.org/jira/browse/HIVE-25317 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.8 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > When we want to use shaded version of hive-exec (i.e., w/o classifier), more > dependencies conflict with Spark. We need to relocate these dependencies too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22738) CVE-2019-0205
[ https://issues.apache.org/jira/browse/HIVE-22738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma resolved HIVE-22738. -- Resolution: Duplicate > CVE-2019-0205 > - > > Key: HIVE-22738 > URL: https://issues.apache.org/jira/browse/HIVE-22738 > Project: Hive > Issue Type: Bug > Components: Thrift API >Reporter: Laurent Goujon >Priority: Major > > There's has been a CVE issued for a Thrift vulnerability which might impact > Hive. The CVE is > [CVE-2019-0205|https://nvd.nist.gov/vuln/detail/CVE-2019-0205], impacts both > clients and servers, and might cause a denial of service through an infinite > loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22294) ConditionalWork cannot be cast to MapredWork When both skew.join and auto.convert is on.
[
https://issues.apache.org/jira/browse/HIVE-22294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415338#comment-17415338
]
Nemon Lou commented on HIVE-22294:
--
The following sql can reproduce this issue,with tpc-ds factor 2, hive 2.3.0:
{code:sql}
use hive_tpcds_text;
set hive.optimize.skewjoin=true;
set hive.auto.convert.join.noconditionaltask.size=1000;
set hive.mapjoin.smalltable.filesize=2500;
select i_item_id,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
from store_sales, customer_demographics, date_dim, item, promotion
where ss_sold_date_sk = d_date_sk and
ss_item_sk = i_item_sk and
ss_cdemo_sk = cd_demo_sk and
ss_promo_sk = p_promo_sk and
cd_gender = 'F' and
cd_marital_status = 'W' and
cd_education_status = 'College' and
(p_channel_email = 'N' or p_channel_event = 'N') and
d_year = 2001
group by i_item_id
order by i_item_id
limit 100;
{code}
Error log:
{noformat}
2021-09-15 10:15:36,602 | ERROR | 43f5fc4c-2294-443e-897e-9c73261d4ccb
HiveServer2-Handler-Pool: Thread-100 | FAILED: ClassCastException
org.apache.hadoop.hive.ql.plan.ConditionalWork cannot be cast to
org.apache.hadoop.hive.ql.plan.MapredWork
java.lang.ClassCastException: org.apache.hadoop.hive.ql.plan.ConditionalWork
cannot be cast to org.apache.hadoop.hive.ql.plan.MapredWork
at
org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:102)
at
org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
at
org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
at
org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:114)
at
org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:292)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11289)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:513)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1318)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1296)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:321)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy38.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:761)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
[jira] [Work logged] (HIVE-24975) Bug in ValidWriteIdList comparison in TxnIdUtils
[
https://issues.apache.org/jira/browse/HIVE-24975?focusedWorklogId=650944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650944
]
ASF GitHub Bot logged work on HIVE-24975:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 06:08
Start Date: 15/Sep/21 06:08
Worklog Time Spent: 10m
Work Description: sourabh912 commented on pull request #2641:
URL: https://github.com/apache/hive/pull/2641#issuecomment-919727850
@kishendas @vihangk1 : I have made a similar fix in Impala
https://github.com/apache/impala/commit/5d307cbb7b2ec3432bebd7759b0bcebf54a6cc22
Please review this and share your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 650944)
Time Spent: 20m (was: 10m)
> Bug in ValidWriteIdList comparison in TxnIdUtils
>
>
> Key: HIVE-24975
> URL: https://issues.apache.org/jira/browse/HIVE-24975
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TxnIdUtils's
> [compare|https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java#L38]
> method returns incorrect result for the following validWriteIdLists
> ValidWriteIdList a = new ValidReaderWriteIdList("default.test:1:1:1:");
> ValidWriteIdList b = new
> ValidReaderWriteIdList("default.test:1:9223372036854775807::");
> TxnIdUtils.compare(a, b) returns +1 whereas the expected response is -1 since
> b is more recent.
> cc - [~kishendas] [~vihangk1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24975) Bug in ValidWriteIdList comparison in TxnIdUtils
[
https://issues.apache.org/jira/browse/HIVE-24975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24975:
--
Labels: pull-request-available (was: )
> Bug in ValidWriteIdList comparison in TxnIdUtils
>
>
> Key: HIVE-24975
> URL: https://issues.apache.org/jira/browse/HIVE-24975
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TxnIdUtils's
> [compare|https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java#L38]
> method returns incorrect result for the following validWriteIdLists
> ValidWriteIdList a = new ValidReaderWriteIdList("default.test:1:1:1:");
> ValidWriteIdList b = new
> ValidReaderWriteIdList("default.test:1:9223372036854775807::");
> TxnIdUtils.compare(a, b) returns +1 whereas the expected response is -1 since
> b is more recent.
> cc - [~kishendas] [~vihangk1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24975) Bug in ValidWriteIdList comparison in TxnIdUtils
[
https://issues.apache.org/jira/browse/HIVE-24975?focusedWorklogId=650943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650943
]
ASF GitHub Bot logged work on HIVE-24975:
-
Author: ASF GitHub Bot
Created on: 15/Sep/21 06:06
Start Date: 15/Sep/21 06:06
Worklog Time Spent: 10m
Work Description: sourabh912 opened a new pull request #2641:
URL: https://github.com/apache/hive/pull/2641
Change-Id: I8022d5725900da08806331ac4d222a47adfe
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 650943)
Remaining Estimate: 0h
Time Spent: 10m
> Bug in ValidWriteIdList comparison in TxnIdUtils
>
>
> Key: HIVE-24975
> URL: https://issues.apache.org/jira/browse/HIVE-24975
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TxnIdUtils's
> [compare|https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java#L38]
> method returns incorrect result for the following validWriteIdLists
> ValidWriteIdList a = new ValidReaderWriteIdList("default.test:1:1:1:");
> ValidWriteIdList b = new
> ValidReaderWriteIdList("default.test:1:9223372036854775807::");
> TxnIdUtils.compare(a, b) returns +1 whereas the expected response is -1 since
> b is more recent.
> cc - [~kishendas] [~vihangk1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
