[
https://issues.apache.org/jira/browse/HIVE-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571791#comment-14571791
]
Rajesh Balamohan commented on HIVE-10551:
-----------------------------------------
[~vikram.dixit] - OOM happens in Map-1 (tried again with apr 29 build
commit:0cad50a193ba777f9271808f057caae674738817. i.e close to the date on which
this JIRA was reported). This happens irrespective of container reuse.
Posting the stacktrace of the OOM (but it could be misleading since it didn't
have enough space to allocate to sorter)
{noformat}
Error: Failure while running task:java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:157)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:345)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.(PipelinedSorter.java:152)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.(PipelinedSorter.java:117)
at
org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:143)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:141)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
... 14 more
{noformat}
> OOM when running query_89 with vectorization on & hybridgrace=false
> -------------------------------------------------------------------
>
> Key: HIVE-10551
> URL: https://issues.apache.org/jira/browse/HIVE-10551
> Project: Hive
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Vikram Dixit K
> Attachments: HIVE-10551-explain-plan.log, hive_10551.png
>
>
> - TPC-DS Query_89 @ 10 TB scale
> - Trunk version of Hive + Tez 0.7.0-SNAPSHOT
> - Additional settings ( "hive.vectorized.groupby.maxentries=1024 ,
> tez.runtime.io.sort.factor=200 tez.runtime.io.sort.mb=1800
> hive.tez.container.size=4096 ,hive.mapjoin.hybridgrace.hashtable=false" )
> Will attach the profiler snapshot asap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
