[
https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-3124:
-----------------------------
Attachment: YARN-3124.3.patch
bq. Merge CapacitySchedulerConfiguration#setCapacitiesByLabels and
CSQueueUtils#setAbsoluteCapacitiesByNodeLabels into a single method
Done
bq. CapacitySchedulerConfiguration#normalizeAccessibleNodeLabels - should
AbstractCSQueue#accessibleLabels be updated as well ?
No, if we have ANY in accessible node labels, and cluster's node label
collection could change, so we need to keep the "ANY" in case any changes of
clusterNodeLabels
bq. why union? newCapacities.getExistingNodeLabels is enough ?
No, this method's semantic is, replace all (absolute)(maximum)capacities, so if
we only use newCapacity.existingNodeLabels, labels which are not existed in new
"ExistingNodeLabels" will not be replaced.
bq. Can the existing get*CapacityByLabel can be removed? use
queueCapacities#get*capacity instead
Done
bq. null for the queueCapacity ? then we can remove the parameter
Updated setQueueConfig logic, see below
bq. remove this?
Done
bq. CSQueueUtils.setAbsoluteCapacitiesByNodeLabel may be inside AbstractCSQueue
Now I consolidate all capacities updating fields to
CSQueueUtils.loadUpdateAndCheckCapacities
bq. QueueCapacities#getExistingNodeLabels - > getNodeLabels?
It seems getExistingNodeLabels is more expressive to me :)
bq. why CSQueueUtils.setAbsoluteCapacitiesByNodeLabels(queueCapacities,
parent); has to be called in ReservationQueue#reinitialize
I added a grand comment in ReservationQueue to explain why we do this
And in beyond, I found we have
ParentQueue/LeafQueue/AbstractCSQueue.setupQueueConfig, there're lots of
parameters we need to maintain, it's not simple when we want to add any new
(configurable) field to queues, I basically removed all parameters in
setupQueueConfig. Instead, all (configurable) fields will be read and
initialized from configuration.
Even if we read the Configuration object twice, but I think it doesn't affect
performance while reinitializing, and thus we can get simpler structure of
queue initialization.
*Attached ver.3 patch*
> Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track
> capacities-by-label
> ------------------------------------------------------------------------------------------------
>
> Key: YARN-3124
> URL: https://issues.apache.org/jira/browse/YARN-3124
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-3124.1.patch, YARN-3124.2.patch, YARN-3124.3.patch
>
>
> After YARN-3098, capacities-by-label (include
> used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be
> tracked in QueueCapacities.
> This patch is targeting to make capacities-by-label in CS Queues are all
> tracked by QueueCapacities.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
