Wangda Tan updated YARN-3124:
    Attachment: YARN-3124.3.patch

bq. Merge CapacitySchedulerConfiguration#setCapacitiesByLabels and 
CSQueueUtils#setAbsoluteCapacitiesByNodeLabels into a single method

bq. CapacitySchedulerConfiguration#normalizeAccessibleNodeLabels - should 
AbstractCSQueue#accessibleLabels be updated as well ?
No, if we have ANY in accessible node labels, and cluster's node label 
collection could change, so we need to keep the "ANY" in case any changes of 

bq. why union? newCapacities.getExistingNodeLabels is enough ?
No, this method's semantic is, replace all (absolute)(maximum)capacities, so if 
we only use newCapacity.existingNodeLabels, labels which are not existed in new 
"ExistingNodeLabels" will not be replaced.

bq. Can the existing get*CapacityByLabel can be removed? use 
queueCapacities#get*capacity instead

bq. null for the queueCapacity ? then we can remove the parameter
Updated setQueueConfig logic, see below

bq. remove this?

bq. CSQueueUtils.setAbsoluteCapacitiesByNodeLabel may be inside AbstractCSQueue
Now I consolidate all capacities updating fields to 

bq. QueueCapacities#getExistingNodeLabels - > getNodeLabels?
It seems getExistingNodeLabels is more expressive to me :)

bq. why CSQueueUtils.setAbsoluteCapacitiesByNodeLabels(queueCapacities, 
parent); has to be called in ReservationQueue#reinitialize
I added a grand comment in ReservationQueue to explain why we do this

And in beyond, I found we have 
ParentQueue/LeafQueue/AbstractCSQueue.setupQueueConfig, there're lots of 
parameters we need to maintain, it's not simple when we want to add any new 
(configurable) field to queues, I basically removed all parameters in 
setupQueueConfig. Instead, all (configurable) fields will be read and 
initialized from configuration.

Even if we read the Configuration object twice, but I think it doesn't affect 
performance while reinitializing, and thus we can get simpler structure of 
queue initialization.

*Attached ver.3 patch*

> Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track 
> capacities-by-label
> ------------------------------------------------------------------------------------------------
>                 Key: YARN-3124
>                 URL: https://issues.apache.org/jira/browse/YARN-3124
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, client, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-3124.1.patch, YARN-3124.2.patch, YARN-3124.3.patch
> After YARN-3098, capacities-by-label (include 
> used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be 
> tracked in QueueCapacities.
> This patch is targeting to make capacities-by-label in CS Queues are all 
> tracked by QueueCapacities.

This message was sent by Atlassian JIRA

Reply via email to