[
https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925767#comment-13925767
]
Jason Lowe commented on MAPREDUCE-5785:
---------------------------------------
Haven't had a chance to look into the patch into great detail, but here are
some initial comments:
- should 'memory.mb.xmx.ratio' be 'memory.mb.heap.ratio'? Even the code names
it that internally. ;-)
- rather than commenting out the mapred-default property it should leave it in
without a value set. See the mapred.child.env entry as an example.
- should be easy to add a unit test that verifies the ratio is working as
intended, e.g.: changing it sees a corresponding jvm argument change out of
MapReduceChildJVM.getVMCommand and setting an explicit heap setting in the
config prevents the ratio from taking effect.
> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mr-am, task
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5785.v01.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.
> One first chooses some container size map reduce.\*.memory.mb and then a
> corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This
> makes sure that the JVM's C-heap (native memory + Java heap) does not exceed
> this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
> - allocating big containers whereas the JVM will only use the default
> -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical
> ratio that can be adjusted. Xmx is not changed automatically if provided by
> the user.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
