[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252919#comment-17252919
]
zhuqi edited comment on YARN-10506 at 12/21/20, 3:30 PM:
---------------------------------------------------------
[~wangda] [~gandras]
When i deep into some test in TestCapacitySchedulerAutoQueueCreation, because
of the update resources now independent of queue init according to YARN-10504
and it will not update policy template related absolute resources and etc
something related, we should finish the policy related absolute update in
ManagedParentQueue's addChildQueue such as :
{code:java}
// In ManagedParentQueue's addChildQueue
for (String label : policy.leafQueueTemplateNodeLabels) {
// Weight will be normalized to queue.weight =
// queue.weight(sum({sibling-queues.weight}))
// When weight is set, capacity will be set to 0;
// When capacity is set, weight will be normalized to 0,
// So get larger from normalized_weight and capacity will make sure we do
// calculation correct
float capacity =
Math.max(policy.leafQueueTemplate.getQueueCapacities().getCapacity(label),
policy.leafQueueTemplate.getQueueCapacities().getNormalizedWeight(label));
if (capacity > 0f) {
policy.leafQueueTemplate.getQueueCapacities()
.setAbsoluteCapacity(label, capacity * (
parentQueueCapacities == null ?
1 :
parentQueueCapacities.getAbsoluteCapacity(label)));
policy.leafQueueTemplateCapacities =
policy.leafQueueTemplate.getQueueCapacities();
}
float maxCapacity =
policy.leafQueueTemplate.getQueueCapacities().getMaximumCapacity(label);
if (maxCapacity > 0f) {
policy.leafQueueTemplate.getQueueCapacities()
.setAbsoluteMaximumCapacity(label, maxCapacity * (
parentQueueCapacities == null ?
1 :
parentQueueCapacities.getAbsoluteMaximumCapacity(label)));
policy.leafQueueTemplateCapacities =
policy.leafQueueTemplate.getQueueCapacities();
}
}
{code}
and policy should update template :
{code:java}
private void updateCapacityFromTemplate(QueueCapacities capacities,
String nodeLabel) {
capacities.setCapacity(nodeLabel,
leafQueueTemplateCapacities.getCapacity(nodeLabel));
capacities.setMaximumCapacity(nodeLabel,
leafQueueTemplateCapacities.getMaximumCapacity(nodeLabel));
capacities.setAbsoluteCapacity(nodeLabel,
leafQueueTemplateCapacities.getAbsoluteCapacity(nodeLabel));
}
{code}
Then the test will pass, but the reinitialize related will still block.
Also when we reinitialize the queue, we should think how to bring up this to
reinitialize and update resource progress.
So i am thinking , if it is worth to make update resources independent of
queue init.
was (Author: zhuqi):
[~wangda] [~gandras]
When i deep into some test in TestCapacitySchedulerAutoQueueCreation, because
of the update resources now independent of queue init according to
[YARN-10504|https://issues.apache.org/jira/browse/YARN-10504], we should finish
the policy related absolute update in ManagedParentQueue's addChildQueue such
as :
{code:java}
// In ManagedParentQueue's addChildQueue
for (String label : policy.leafQueueTemplateNodeLabels) {
// Weight will be normalized to queue.weight =
// queue.weight(sum({sibling-queues.weight}))
// When weight is set, capacity will be set to 0;
// When capacity is set, weight will be normalized to 0,
// So get larger from normalized_weight and capacity will make sure we do
// calculation correct
float capacity =
Math.max(policy.leafQueueTemplate.getQueueCapacities().getCapacity(label),
policy.leafQueueTemplate.getQueueCapacities().getNormalizedWeight(label));
if (capacity > 0f) {
policy.leafQueueTemplate.getQueueCapacities()
.setAbsoluteCapacity(label, capacity * (
parentQueueCapacities == null ?
1 :
parentQueueCapacities.getAbsoluteCapacity(label)));
policy.leafQueueTemplateCapacities =
policy.leafQueueTemplate.getQueueCapacities();
}
float maxCapacity =
policy.leafQueueTemplate.getQueueCapacities().getMaximumCapacity(label);
if (maxCapacity > 0f) {
policy.leafQueueTemplate.getQueueCapacities()
.setAbsoluteMaximumCapacity(label, maxCapacity * (
parentQueueCapacities == null ?
1 :
parentQueueCapacities.getAbsoluteMaximumCapacity(label)));
policy.leafQueueTemplateCapacities =
policy.leafQueueTemplate.getQueueCapacities();
}
}
{code}
and policy should update template :
{code:java}
private void updateCapacityFromTemplate(QueueCapacities capacities,
String nodeLabel) {
capacities.setCapacity(nodeLabel,
leafQueueTemplateCapacities.getCapacity(nodeLabel));
capacities.setMaximumCapacity(nodeLabel,
leafQueueTemplateCapacities.getMaximumCapacity(nodeLabel));
capacities.setAbsoluteCapacity(nodeLabel,
leafQueueTemplateCapacities.getAbsoluteCapacity(nodeLabel));
}
{code}
Then the test will pass, but the reinitialize related will block.
Also when we reinitialize the queue, we should think how to bring up this to
reinitialize and update resource progress.
So i am thinking , if it is worth to make update resources independent of
queue init.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10506.001.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
