[
https://issues.apache.org/jira/browse/HIVE-12547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042073#comment-15042073
]
Sergey Shelukhin commented on HIVE-12547:
-----------------------------------------
Why? Otherwise, we get query failures and the only workaround is to reduce the
conversion thhreshold, which doesn't map to anything in an obvious way, so it's
easy to disable mapjoin for queries that would benefit from it.
> VectorMapJoinFastBytesHashTable fails during expansion
> ------------------------------------------------------
>
> Key: HIVE-12547
> URL: https://issues.apache.org/jira/browse/HIVE-12547
> Project: Hive
> Issue Type: Bug
> Reporter: Siddharth Seth
> Assignee: Sergey Shelukhin
> Priority: Critical
>
> {code}
> 2015-11-30 20:55:30,361 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1448429572030_1224_7][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 2, taskAttemptId=attempt_1448429572030_1224_7_03_000005_0,
> creationTime=1448934722881, allocationTime=1448934726552,
> startTime=1448934726553, finishTime=1448934930360, timeTaken=203807,
> status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while
> running task:
> attempt_1448429572030_1224_7_03_000005_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:289)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:424)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:394)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:519)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:472)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:274)
> ... 15 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:414)
> ... 20 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:106)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:131)
> ... 4 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:110)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:293)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:174)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:170)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:104)
> ... 5 more
> Caused by: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.expandAndRehash(VectorMapJoinFastBytesHashTable.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.add(VectorMapJoinFastBytesHashTable.java:73)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.putRow(VectorMapJoinFastBytesHashTable.java:53)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.putRow(VectorMapJoinFastTableContainer.java:182)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:97)
> ... 9 more
> {code}
> Array size wraps around integer limits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
