[
https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728333#comment-13728333
]
Sandy Ryza commented on MAPREDUCE-5311:
---------------------------------------
SLOTS_MILLIS_MAPS is a confusing property that I think we should remove if at
all possible. Independent of YARN-1004, the current way of calculating it is a
hack that neither adheres to its behavior in MR1 nor its description.
In MR1, it was calculated as the sum of all map task's running times multiplied
by the number of slots each task took up.
The description, "Total time spent by all maps in occupied slots (ms)", with
its unit being ms and not slots-ms, reads to me as a sum of all map task's
running times, independent of the number of slots consumed by each.
On most clusters, however, these were equivalent, because a map task would only
take up more than one slot if mapred.cluster.map.memory.mb were set to
something other than its default, -1.
The MR2 counter, which is calculated as the sum of all map task's running times
multiplied by the memory allocated by each one, divided by the minimum
allocation, adheres to neither of these. The average MR1 user who did not set
mapred.cluster.map.memory.mb will find, upon upgrading the MR2, that
SLOTS_MILLIS_MAPS functions differently than it did in MR1, as well as be
confused by YARN's claim that it gets rid of slots.
This is not an obscure feature. It's one of the things that any person running
a MapReduce job will see, and I think keeping it in its current form will help
to confuse those who are trying to upgrade to MR2 and understand the new
resource model.
> Remove slot millis computation logic and deprecate counter constants
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-5311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 2.0.4-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch,
> MAPREDUCE-5311.patch
>
>
> Per discussion in MAPREDUCE-5310 and comments in the code we should remove
> all the related logic and just leave the counter constant for backwards
> compatibility and deprecate the counter constants.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
