[
https://issues.apache.org/jira/browse/TEZ-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106468#comment-14106468
]
Siddharth Seth commented on TEZ-1473:
-------------------------------------
{code}
<property>
<name>tez.task.launch.cmd-opts</name>
<value>-server -Xmx4096m -Djava.net.preferIPv4Stack=true -XX:+UseNUMA
-XX:+UseParallelGC -Dhadoop.metrics.log.level=WARN</value>
</property>
<property>
<name>tez.task.resource.memory.mb</name>
<value>4096</value>
{code}
Xmx should always be less than "tez.task.resource.memory.mb". The Xmx value
should always be lower than the tez.task.resource.memory.mb (similarly for the
AM option). Xmx is just the java heap, the process itself will use additional
memory beyond this. In case a large chunk of the heap ends up getting used -
the YARN PMEM killer will kill the task.
You could skip the Xmx option completely in this case. Tez (0.5) will default
this to 80% of the tez.task.resource.memory.mb
3KB vs 8KB on tez.runtime.shuffle.buffersize should likely not make a lot of
difference (unless a lot of these are created in parallel - which I don't
believe is the case). That said, I have seen memory issues setting
io.file.buffer.size to a large value. There shouldn't be too many of those
buffers either. cc/[~gopalv]
> TEZ_RUNTIME_SHUFFLE_BUFFER is too large by default
> --------------------------------------------------
>
> Key: TEZ-1473
> URL: https://issues.apache.org/jira/browse/TEZ-1473
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Tsuyoshi OZAWA
> Assignee: Tsuyoshi OZAWA
> Attachments: TEZ-1473.1.patch
>
>
> TEZ_RUNTIME_SHUFFLE_BUFFER is 8GB by default, while
> TEZ_TASK_RESOURCE_MEMORY_MB_DEFAULT is 1GB. It leads OoM or Container Killer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
