[
https://issues.apache.org/jira/browse/TEZ-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893531#comment-15893531
]
Siddharth Seth commented on TEZ-3647:
-------------------------------------
[~aplusplus] - can we change this to work based on the old config parameter
itself? We don't need more configs for this change.
TezConfiguration.TEZ_CONTAINER_MAX_JAVA_HEAP_FRACTION
Set this to -1, and Tez makes some determinations on the Xmx value. This is the
same as the defaults you have in the patch.
Eventually we can move to this being completely pluggable where users can
inspect opts, GC parameters, java version?, container size etc - and come up
with the final parameters.
> Lower default Xmx for small AM container
> -----------------------------------------
>
> Key: TEZ-3647
> URL: https://issues.apache.org/jira/browse/TEZ-3647
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Zhiyuan Yang
> Assignee: Zhiyuan Yang
> Attachments: TEZ-3647.1.patch
>
>
> The default Xmx is 80% of container size. For a 2G container, non-heap memory
> can cost more than 20%. Total off-heap memory is around 450MB. In case of
> nearly full heap memory usage, we will be out of physical memory and killed
> by YARN.
> Here is native memory summary for a AM running tpcds query 13.
> {code}
> Native Memory Tracking:
> Total: reserved=3424708KB, committed=2155712KB
> - Java Heap (reserved=1677312KB, committed=1677312KB)
> (mmap: reserved=1677312KB, committed=1677312KB)
> - Class (reserved=1096701KB, committed=54061KB)
> (classes #8465)
> (malloc=1021KB #11433)
> (mmap: reserved=1095680KB, committed=53040KB)
> - Thread (reserved=195103KB, committed=195103KB)
> (thread #191)
> (stack: reserved=193948KB, committed=193948KB)
> (malloc=611KB #986)
> (arena=543KB #382)
> - Code (reserved=254302KB, committed=31222KB)
> (malloc=4702KB #7076)
> (mmap: reserved=249600KB, committed=26520KB)
> - GC (reserved=148963KB, committed=148963KB)
> (malloc=53951KB #17562)
> (mmap: reserved=95012KB, committed=95012KB)
> - Compiler (reserved=306KB, committed=306KB)
> (malloc=176KB #311)
> (arena=131KB #3)
> - Internal (reserved=17872KB, committed=17872KB)
> (malloc=17840KB #15275)
> (mmap: reserved=32KB, committed=32KB)
> - Symbol (reserved=12659KB, committed=12659KB)
> (malloc=10829KB #106221)
> (arena=1830KB #1)
> - Native Memory Tracking (reserved=2735KB, committed=2735KB)
> (malloc=192KB #2968)
> (tracking overhead=2543KB)
> - Arena Chunk (reserved=5034KB, committed=5034KB)
> (malloc=5034KB)
> - Tracing (reserved=10444KB, committed=10444KB)
> (malloc=10444KB #274)
> - Unknown (reserved=3276KB, committed=0KB)
> (mmap: reserved=3276KB, committed=0KB)
> {code}
> Please notice the memory here means virtual memory size. In worst scenario
> all pages get loaded and takes same amount of physical memory.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
