[
https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221139#comment-14221139
]
Karthik Kambatla commented on MAPREDUCE-5785:
---------------------------------------------
Thanks for the review, Sandy. Addressed most of your comments.
bq. It looks like this value will be overwritten in all cases.
Actually no. We use an if - else if without an else.
bq. This is a little weird. Can we assign heapSize outside of the condition?
I changed this in the updated patch, but this was to avoid parsing the heapSize
unnecessarily and complicating the code with more if-else's.
bq. Indentation here is inconsistent with other places.
The file has different kinds of indentation. I would like to leave this as is,
if you are not particular about it.
bq. Any reason to have getTaskJavaOpts in JobConf instead of MapReduceChildJVM?
MapReduceChildJVM needs the javaopts and TaskAttemptImpl needs the memory. We
could keep those two methods closer to the access points, but I felt the two
methods should be closer together than to the access points.
> Derive task attempt JVM max heap size and io.sort.mb automatically from
> mapreduce.*.memory.mb
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mr-am, task
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch,
> MAPREDUCE-5785.v03.patch, mr-5785-4.patch, mr-5785-5.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.
> One first chooses some container size map reduce.\*.memory.mb and then a
> corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This
> makes sure that the JVM's C-heap (native memory + Java heap) does not exceed
> this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
> - allocating big containers whereas the JVM will only use the default
> -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical
> ratio that can be adjusted. Xmx is not changed automatically if provided by
> the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
