[jira] [Commented] (HIVE-22662) java.lang.NegativeArraySizeException when trying to get HashPartition
[
https://issues.apache.org/jira/browse/HIVE-22662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009745#comment-17009745
]
Diego Patricio Moncayo commented on HIVE-22662:
---
Sorry for the late answer. I confirm that with the change in the parameter, the
error does not happen. Although some OOM appeared in other queries.
Thank you [~gopalv]
> java.lang.NegativeArraySizeException when trying to get HashPartition
> -
>
> Key: HIVE-22662
> URL: https://issues.apache.org/jira/browse/HIVE-22662
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
> Environment: Using hdp 3.1.4.0-315
>Reporter: Diego Patricio Moncayo
>Priority: Minor
>
> Running a query between a big table and a small one, this error appeared:
> {code:java}
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:361)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async
> Initialization failed. abortRequested=false
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:465)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:399)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:572)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:524)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:342)
> ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:103)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:271)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96)
> ... 5 more
> Caused by: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.(HybridHashTableContainer.java:362)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.(HybridHashTableContainer.java:277)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:231)
> ... 9 more
> {code}
> When looking into the stacktrace, this line appeared.
> {code:java}
> hashPartitions = new HashPartition[numPartitions];
> {code}
> Apparently the int variable numPartition overflows
> When this parameter is set hive.auto.convert.join=false the issue disappears
> Similar to this Issue https://issues.apache.org/jira/browse/HIVE-12547 but
> not the same
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22662) java.lang.NegativeArraySizeException when trying to get HashPartition
[
https://issues.apache.org/jira/browse/HIVE-22662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999340#comment-16999340
]
Gopal Vijayaraghavan commented on HIVE-22662:
-
HybridHashTableContainer was disabled by default in HiveConf
{code}
HIVEUSEHYBRIDGRACEHASHJOIN("hive.mapjoin.hybridgrace.hashtable", false,
"Whether to use hybrid" +
"grace hash join as the join method for mapjoin. Tez only."),
{code}
bq. this parameter is set hive.auto.convert.join=false the issue disappears
Can you confirm what happens when the hybridgrace is turned off to match the
main branch features?
> java.lang.NegativeArraySizeException when trying to get HashPartition
> -
>
> Key: HIVE-22662
> URL: https://issues.apache.org/jira/browse/HIVE-22662
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
> Environment: Using hdp 3.1.4.0-315
>Reporter: Diego Patricio Moncayo
>Priority: Minor
>
> Running a query between a big table and a small one, this error appeared:
> {code:java}
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:361)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async
> Initialization failed. abortRequested=false
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:465)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:399)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:572)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:524)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:342)
> ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:103)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:271)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96)
> ... 5 more
> Caused by: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.(HybridHashTableContainer.java:362)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.(HybridHashTableContainer.java:277)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:231)
> ... 9 more
> {code}
> When looking into the stacktrace, this line appeared.
> {code:java}
> hashPartitions = new HashPartition[numPartitions];
> {code}
> Apparently the int variable numPartition overflows
> When this parameter is set hive.auto.convert.join=false the issue disappears
> Similar to this Issue https://issues.apache.org/jira/browse/HIVE-12547 but
> not the same
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
