[
https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190842#comment-13190842
]
Thomas Graves commented on MAPREDUCE-3681:
------------------------------------------
Thanks for the update Arun.
I was misunderstanding what usedCapacity meant, but I think I now understand,
although I don't think the current formula for used Capacity works in all
cases. Please correct me if I’m off on the definition here. I believe used
Capacity is meant to be what % of the parent am I using. Similar to how
capacity/max capacity apply to the parent. I expected usedCapacity to be what
utilization is defined as, which is what I thought users will be interested in
-> how much of the resources normally allocated to my queue (absoluteCapacity)
am I using right now. I think Utilization = absoluteUsedCapacity.
Is a user normally really going to care what % of my parent queue they are
using? To me the absoluteCapacity is what I’m interested in because my queue
has X amount of memory allocated to it, and I have Y left.
I can see admins being interested in the used Capacity as they can quickly see
what percent of the parent is being used by the various leaf queues.
right now definitions are:
utilization = usedResources.getMemory() / (clusterResource.getMemory() *
absoluteCapacity)
usedCapacity = usedResources.getMemory() / (clusterResource.getMemory() *
capacity)
Here is an example, which I believe demonstrates usedCapacity being off - lets
say cluster has 32G memory total:
Queue definitions:
root - capacity = absoluteCapacity = 100% - 32G
root.a - capacity=50%, absoluteCap = 100*50=50% - 16G
root.a.a1 - capacity = 50%, absoluteCap = (100*50)*50 = 25% - 8G
root.a.a2 - capacity = 50%, absoluteCap = (100*50)*50% = 25% - 8G
root.b - capacity = 20%, absoluteCap = 100*20=20% - 6.4G
root.b.b1 - capacity = 60%, absoluteCap = 100*20%*60% = 12%, MaxCap = -1 (lets
me go over my capacity) = 3.84G
root.c - capacity = 30%, absoluteCap = 100*30=30% - 9.6G
root.c.c1 - capacity = 75%, absoluteCap = 100*30%*75* = 22.5% = 7.2G
Now I start a job that uses 4G submitted to a1.
root used = 4/(32 * 1) = 12.5%, utilization= 4/(32*1) = 12.5% = 4G of 32G
root.a used = 4/(32*.5) = 25%, utilization = 4/(32*.5) = 25% = 4G of 16G
root.a.a1 used = 4/(32 *.5) = 25%, utilization = 4/(32*.25) = 50% => this just
happens to work because capacity = parent asoluteCap
job in a1 finishes. now start a job that uses 4G submitted to b1
root used = 4/(32 * 1) = 12.5%, utilization= 4/(32*1) = 12.5% = 4G of 32G
root.b used = 4/(32*.2) = 62.5%, utilization = 4/(32*.2) = 62.5% = 4G of 6.4G
root.b.b1 used = 4/(32 *.6) = 20.8% (4G of 6.4G ??), utilization = 4/(32*.12) =
104% (4G of 3.84G)
the 20.8% doesn't make sense to me? I'm using 4G of my parents 6.4G - that
should be 62.5%.
job in a1 finishes. now start a job that uses 4G submitted to c1
root used = 4/(32 * 1) = 12.5%, utilization= 4/(32*1) = 12.5% = 4G of 32G
root.c used = 4/(32*.3) = 41.6%, utilization = 4/(32*.3) = 41.6% = 4G of 9.6G
root.c.c1 used = 4/(32 *.75) = 16.6% (4G of 9.6G ??), utilization = 4/(32*.225)
= 55.5% (4G of 7.2G)
Again the 16.6% I think is wrong as it should be 4G of the 9.6G or 41.6%
I think the formula should be
usedCapacity = usedResources.getMemory() / (clusterResource.getMemory() *
parent.getAbsoluteCapacity())
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical
> where you have root -> parent queue -> leaf queue, the leaf queue doesn't
> calculate the used capacity properly. It seems to be using the entire cluster
> memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like
> getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
