[
https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269888#comment-17269888
]
zhuqi commented on YARN-10590:
------------------------------
cc [~leftnoteasy] [~sunilg] [~bteke]
When i deep into the loss of the caculation about
{code:java}
// TODO: Wangda, I think this is a wrong test, it doesn't consider rounding
// loss of multiplication, the right value should be <10240, 2>, but the
// test expects <10240, 1>
{code}
I think the test logic should change to :
{code:java}
// When in absolute mode.
// We can only assertEquals in memory size, because the cap/absolute
// is caculated by memory, we can only make sure this.
if (leafQueue.getCapacityConfigType() ==
AbstractCSQueue.CapacityConfigType.ABSOLUTE_RESOURCE) {
assertEquals(effMinCapacity.getMemorySize(),
leafQueue.getEffectiveCapacity(label).getMemorySize());
assertTrue(effMinCapacity.getVirtualCores()
<= leafQueue.getEffectiveCapacity(label).getVirtualCores());
} else {
assertEquals(effMinCapacity, leafQueue.getEffectiveCapacity(label));
}
{code}
Because in absolute mode with auto created leaf queue, we get the cap/absolute
by :
{code:java}
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
....
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
...
}
}
{code}
In default resource caculator, we get the cap/absolute only by memory:
{code:java}
public float ratio(Resource a, Resource b) {
return divideSafelyAsFloat(a.getMemorySize(), b.getMemorySize());
}
{code}
This cause the loss of multiply by cap/absolute, with the actual effective min
resource.
The actual effective min resource will be updated in :
{code:java}
// Update effective resource (min/max) to each child queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
}{code}
We will get the actual effective min resource, different with the get the
cap/absolute only by memory.
So in absolute auto created leaf queue , the effective min resource only can
meet the memory caculated by cap/absolute with default resource caculator.
The logic is wright, but the test we should fix.
Thoughts?
Thanks.
> Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute
> caculation loss.
> -------------------------------------------------------------------------------------
>
> Key: YARN-10590
> URL: https://issues.apache.org/jira/browse/YARN-10590
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: zhuqi
> Assignee: zhuqi
> Priority: Major
> Attachments: YARN-10590.001.patch
>
>
> Because we have introduced YARN-10504.
> In auto created leaf queue with absolute mode, the caculation of effective
> min resource will different from other mode.
> We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
