[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259772#comment-17259772
]
Andras Gyori commented on YARN-10504:
-------------------------------------
Fixed the following test classes:
* TestCSMaxRunningAppsEnforcer
* TestNodeLabelContainerAllocation
* TestQueueParsing
* TestCapacitySchedulerDynamicBehavior
* TestCapacitySchedulerPlanFollower
I have identified a problematic part of the change, which must be addressed,
because it affects a lot of classes.
The *CSQueueUtils* has been changed and now only the
*loadCapacitiesByLabelsFromConf* method is used. Several tests are failing
because:
{code:java}
float absCapacity = queueCapacities.getCapacity(label);
float absMaxCapacity = queueCapacities.getMaximumCapacity(label);
if (absCapacity > absMaxCapacity) {
throw new IllegalArgumentException("Illegal queue capacity setting "
+ "(abs-capacity=" + absCapacity + ") > (abs-maximum-capacity="
+ absMaxCapacity + ") for queue=["
+ queuePath + "],label=[" + label + "]");
}{code}
Now, it might be a typo or a bug, because absCapacity here is NOT the actual
absolute capacity, but the regular capacity. This leads to an other problem,
namely the absolute capacity is not calculated for LeafQueue and PlanQueue,
which leaves the absolute capacity as 0. This was previously calculated in
*CSQueueUtils*. The *AbstractCSQueue*#*updateAbsoluteCapacity* contains the
same logic, but the AutoCreatedLeafQueue tests are still failing, and probably
due to this issue.
The questions are:
* Is the check in loadCapacitiesByLabelsFromConf correct, or absCapacity
should be equals to queueCapacities.getAbsolutCapacity()
* Where do we calculate the absolute capacity, which is not handled now in two
queue classes?
> Implement weight mode in Capacity Scheduler
> -------------------------------------------
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Benjamin Teke
> Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
