[
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067012#comment-16067012
]
Robert Kanter commented on MAPREDUCE-6647:
------------------------------------------
I think [~haibo.chen]'s example supports your use case then. If the minimum
allocation is 2GB but the user requests 1GB, Yarn will give them 2GB but prior
to MR-6647, they'd only be charged for 1GB, which is incorrect.
> MR usage counters use the resources requested instead of the resources
> allocated
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch,
> mapreduce6647.003.patch, mapreduce6647.004.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the
> resources requested instead of the resources allocated. The scheduler
> increment-allocation-mb configs could lead to these values not being the
> same. We could change the counters to use the allocated resources in order to
> account for this.
> {code}
> private static void updateMillisCounters(JobCounterUpdateEvent jce,
> TaskAttemptImpl taskAttempt) {
> /***omitted**/
> long duration = (taskAttempt.getFinishTime() -
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf,
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
> YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
> YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired *
> duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired
> * duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration *
> mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
