[
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063865#comment-16063865
]
Vrushali C commented on MAPREDUCE-6647:
---------------------------------------
Hi [~haibo.chen] [~rkanter]
Although it's technically a correct use case that we need a counter that
reports what the actual usage is, this update will break the meaning for
downstream apps which have been using these counter to track their usage. They
could now suddenly see a "drop" in their usage of memory or vcores or both.
Also, how do you envision tracking the resource allocation for these jobs? Is
there another counter that tells us how much memory was reserved for this job
on the cluster (which is what mbmillis meant before this patch)? Similarly for
vcore-millis? Because that memory was not available for scheduling other
containers.
> MR usage counters use the resources requested instead of the resources
> allocated
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch,
> mapreduce6647.003.patch, mapreduce6647.004.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the
> resources requested instead of the resources allocated. The scheduler
> increment-allocation-mb configs could lead to these values not being the
> same. We could change the counters to use the allocated resources in order to
> account for this.
> {code}
> private static void updateMillisCounters(JobCounterUpdateEvent jce,
> TaskAttemptImpl taskAttempt) {
> /***omitted**/
> long duration = (taskAttempt.getFinishTime() -
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf,
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
> YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
> YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired *
> duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired
> * duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration *
> mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
