[
https://issues.apache.org/jira/browse/HADOOP-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637871#action_12637871
]
Hemanth Yamijala commented on HADOOP-4373:
------------------------------------------
The observed behavior is because we compute the guaranteed capacity as follows:
{code}
private synchronized void updateQSIObjects() {
int slotsDiff = getClusterCapacity()- numSlots;
numSlots += slotsDiff;
for (QueueSchedulingInfo qsi: queueInfoMap.values()) {
// compute new GCs and ACs, if TT slots have changed
if (slotsDiff != 0) {
qsi.guaranteedCapacity +=
(qsi.guaranteedCapacityPercent*slotsDiff/100);
}
//...
}
{code}
The intent, it appears, is that after a certain steady state is reached, i.e.
after the time the guaranteed capacities for queues reaches some % of the total
capacity, guaranteed capacities are updated by a differential based on the
difference in slots. While this seems right, the problem is that the
pre-condition never occurs.
{{getClusterCapacity()}} gets the number of map/reduce slots currently known.
When the cluster is starting up, this value gradually increases. Typically,
{{slotsDiff}} will be equal to the number of slots on a TT, as each TT joins
in. This, in turn, is typically much less than 100 (say 2, or 4, or some such).
So, if the {{guaranteedCapacityPercent}} is a small value as well, something
like 10% or so, the product is < 100, and so the capacity is always 0.
Trying out a patch which modifies the computation as follows:
{code}
if (slotsDiff != 0) {
qsi.guaranteedCapacity =
(int)(qsi.guaranteedCapacityPercent*numSlots/100);
}
{code}
i.e. if there is a difference in slots, we don't compute the differential, but
rather the actual capacity. Does this seem right ?
> Guaranteed Capacity calculation is not calculated correctly
> -----------------------------------------------------------
>
> Key: HADOOP-4373
> URL: https://issues.apache.org/jira/browse/HADOOP-4373
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Karam Singh
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Guaranteed Capacity calculation is not calculated correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
