[
https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215266#comment-14215266
]
Karthik Kambatla commented on MAPREDUCE-5785:
---------------------------------------------
Sorry for the dropping the ball on this. Few comments:
# For those moving from MR1 to MR2, specifying Xmx might be easier. In those
cases, it would be nice to estimate the memory.mb values. I am open to doing
this in another JIRA, and would be happy to work on it and post a patch.
# Multiplying container-size by 0.8 to arrive at the heapsize does read better.
We will have to divide by 0.8 to do it in the other direction.
# In the patch itself, deprecating mapred.child.java.opts is fine but I would
refer to .map.java.opts and .reduce.java.opts instead of the container size
values. Those properties instead could point to the memory.mb properties.
I understand it has been a while since we last looked at this. [~jira.shegalov]
- will you be able to address these comments? I can help with addressing the
comments if you are otherwise caught up.
> Derive task attempt JVM max heap size and io.sort.mb automatically from
> mapreduce.*.memory.mb
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mr-am, task
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch,
> MAPREDUCE-5785.v03.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.
> One first chooses some container size map reduce.\*.memory.mb and then a
> corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This
> makes sure that the JVM's C-heap (native memory + Java heap) does not exceed
> this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
> - allocating big containers whereas the JVM will only use the default
> -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical
> ratio that can be adjusted. Xmx is not changed automatically if provided by
> the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
