[
https://issues.apache.org/jira/browse/HIVE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936035#comment-14936035
]
Gopal V commented on HIVE-11355:
--------------------------------
[~vikram.dixit]: the feature seems to over-estimate sorter sizes larger than
the oldgen sizes in the JVM, the Xmx is 80% of the container size & the goal of
this is to only scale down the buffers from their configured size.
I noticed that it occasionally, the decider decides to scale it upwards to bad
results.
{code}
], TaskAttempt 3 failed, info=[Error: Failure while running task:
attempt_1442254312093_1019_1_00_000016_3:java.lang.IllegalArgumentException:
tez.runtime.io.sort.mb 8187 should be larger than 0 and should be less than the
available task memory (MB):6311
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.getInitialMemoryRequirement(ExternalSorter.java:338)
at
org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.initialize(OrderedPartitionedKVOutput.java:92)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:477)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:455)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
To repro, run query28 on 30Tb scale (planner on cn105).
> Hive on tez: memory manager for sort buffers (input/output) and operators
> -------------------------------------------------------------------------
>
> Key: HIVE-11355
> URL: https://issues.apache.org/jira/browse/HIVE-11355
> Project: Hive
> Issue Type: Improvement
> Components: Tez
> Affects Versions: 2.0.0
> Reporter: Vikram Dixit K
> Assignee: Vikram Dixit K
> Attachments: HIVE-11355.1.patch, HIVE-11355.2.patch,
> HIVE-11355.3.patch, HIVE-11355.4.patch, HIVE-11355.5.patch
>
>
> We need to better manage the sort buffer allocations to ensure better
> performance. Also, we need to provide configurations to certain operators to
> stay within memory limits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
