[
https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925720#comment-13925720
]
Gera Shegalov commented on MAPREDUCE-5785:
------------------------------------------
[~kkambatl], the constant headroom has been discussed. My thinking in favor of
percentage-kind of overhead is that
- it's easy to reason about direct proportional overhead
- reasons a larger container size is specified is to process more data. Side
effects of it is that some code is executed more frequently and more byte code
is compiled into native. The more native memory is used through the NIO stack,
and native compressor libraries. The more tracking structures a GC might have.
I like your idea to tune io.sort.mb accordingly. I'd pick the default 50% of
Xmx to match the current defaults: io.sort.mb=100 and -Xmx200m. I'll add this
to the patch.
> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mr-am, task
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5785.v01.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.
> One first chooses some container size map reduce.\*.memory.mb and then a
> corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This
> makes sure that the JVM's C-heap (native memory + Java heap) does not exceed
> this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
> - allocating big containers whereas the JVM will only use the default
> -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical
> ratio that can be adjusted. Xmx is not changed automatically if provided by
> the user.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
