[
https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738908#comment-13738908
]
Jason Lowe commented on MAPREDUCE-5311:
---------------------------------------
bq. Defining SLOTS_MILLIS to be the same as CONTAINER_MILLIS will break
compatibility for the minority of clusters that both used the
mapred.cluster.map.memory.mb property in MR1 and chose to configure memory in
MR2 in a way that exactly lines up with slots in MR1 (i.e. setting
yarn.scheduler.minimum-allocation-mb to exactly the same as
mapred.cluster.map.memory.mb, which is often not the right choice).
It will also break compatibility for those on 0.23 where SLOT_MILLIS is
currently based on the minimum allocation. Changing the units from slots to
containers for those cases will significantly lower the reported value and
could be misinterpreted as lower cluster utilization. That's why I'm not in
favor of changing the SLOT_MILLIS units but still calling it SLOT_MILLIS. I'd
rather the counter disappear than have it be a completely wrong value. As Arun
mentioned, there are in-house tools to compute cluster utilization based on
these metrics.
I agree that supporting SLOT_MILLIS long-term is problematic due to the removal
of the minimum or slot concept, hence my earlier suggestion for something more
supportable like MEM_MILLIS or CPU_MILLIS. For backwards compatibility, we
could consider doing something along the lines of what it's already doing
today, i.e.: having the AM read a config and computing the "slot" size from
there. That way admins can configure whatever is appropriate for a "slot" for
their cluster to keep legacy tools reporting something appropriate when they
try to look for SLOT_MILLIS. If the property is not configured then arguably
the AM should not report a SLOT_MILLIS job counter since it doesn't know how
big a slot should be.
> Replace SLOTS_MILLIS counters with MEM_MILLIS
> ---------------------------------------------
>
> Key: MAPREDUCE-5311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 2.0.4-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Sandy Ryza
> Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch,
> MAPREDUCE-5311.patch
>
>
> Per discussion in MAPREDUCE-5310 and comments in the code we should remove
> all the related logic and just leave the counter constant for backwards
> compatibility and deprecate the counter constants.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
