[
https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925159#comment-13925159
]
Sandy Ryza commented on MAPREDUCE-5785:
---------------------------------------
Something like this has been long-needed.
Though I'm worried that it's not backwards-compatible - users would see their
JVM max heaps change in certain situations. In situations where they didn't
set the max heap, were cutting it close, but were still OK, they could see
OutOfMemoryErrors after the change.
Another thing is that, as a user, I care more about my max heap size than how
much I request from YARN. The latter is usually a consequence of the former.
One possible way around both of these would be to add a new parameter that
controls max heap size and sets mapreduce.*.memory.mb accordingly.
> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mr-am, task
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5785.v01.patch
>
>
> Currently users have to set 2 memory-related configs for per Job / per task
> type. One fist choses some container size mapreduce.*.memory.mb and then a
> corresponding Xmx < mapreduce.*.memory.mb to make sure that the JVM with the
> user code heap, and its native memory do not exceed this limit. If one
> forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM
> will only use the default -Xmx200m.
> With this JIRA, we propose to set Xmx automatically base on an empirical
> ratio that can be adjusted. Xmx is not changed automaically if provided by
> the user.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
