[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-3124: -- Attachment: YARN-3124-branch-2.6.1.txt Attaching patch that I committed to 2.6.1. > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-3124-branch-2.6.1.txt, YARN-3124.1.patch, > YARN-3124.2.patch, YARN-3124.3.patch, YARN-3124.4.patch, YARN-3124.5.patch > > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-3124: -- Labels: 2.6.1-candidate (was: ) Fix Version/s: 2.6.1 Pulled this into 2.6.1 as a dependency. Had to fix merge conflicts and make minor changes to get it to work on 2.6.1. Ran all ResourceManager tests before pushing this in. > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-3124.1.patch, YARN-3124.2.patch, YARN-3124.3.patch, > YARN-3124.4.patch, YARN-3124.5.patch > > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3124: - Attachment: YARN-3124.5.patch Thanks comments [~jianhe]. Attached new patch (ver.5) > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3124.1.patch, YARN-3124.2.patch, YARN-3124.3.patch, > YARN-3124.4.patch, YARN-3124.5.patch > > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3124: - Attachment: YARN-3124.4.patch Fixed findbugs warning, attached ver.4 patch > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3124.1.patch, YARN-3124.2.patch, YARN-3124.3.patch, > YARN-3124.4.patch > > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3124: - Attachment: YARN-3124.3.patch bq. Merge CapacitySchedulerConfiguration#setCapacitiesByLabels and CSQueueUtils#setAbsoluteCapacitiesByNodeLabels into a single method Done bq. CapacitySchedulerConfiguration#normalizeAccessibleNodeLabels - should AbstractCSQueue#accessibleLabels be updated as well ? No, if we have ANY in accessible node labels, and cluster's node label collection could change, so we need to keep the "ANY" in case any changes of clusterNodeLabels bq. why union? newCapacities.getExistingNodeLabels is enough ? No, this method's semantic is, replace all (absolute)(maximum)capacities, so if we only use newCapacity.existingNodeLabels, labels which are not existed in new "ExistingNodeLabels" will not be replaced. bq. Can the existing get*CapacityByLabel can be removed? use queueCapacities#get*capacity instead Done bq. null for the queueCapacity ? then we can remove the parameter Updated setQueueConfig logic, see below bq. remove this? Done bq. CSQueueUtils.setAbsoluteCapacitiesByNodeLabel may be inside AbstractCSQueue Now I consolidate all capacities updating fields to CSQueueUtils.loadUpdateAndCheckCapacities bq. QueueCapacities#getExistingNodeLabels - > getNodeLabels? It seems getExistingNodeLabels is more expressive to me :) bq. why CSQueueUtils.setAbsoluteCapacitiesByNodeLabels(queueCapacities, parent); has to be called in ReservationQueue#reinitialize I added a grand comment in ReservationQueue to explain why we do this And in beyond, I found we have ParentQueue/LeafQueue/AbstractCSQueue.setupQueueConfig, there're lots of parameters we need to maintain, it's not simple when we want to add any new (configurable) field to queues, I basically removed all parameters in setupQueueConfig. Instead, all (configurable) fields will be read and initialized from configuration. Even if we read the Configuration object twice, but I think it doesn't affect performance while reinitializing, and thus we can get simpler structure of queue initialization. *Attached ver.3 patch* > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3124.1.patch, YARN-3124.2.patch, YARN-3124.3.patch > > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3124: - Attachment: YARN-3124.2.patch Rebased against trunk. > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3124.1.patch, YARN-3124.2.patch > > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3124: - Attachment: YARN-3124.1.patch Attached ver.1 patch > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3124.1.patch > > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
[ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3124: - Description: After YARN-3098, capacities-by-label (include used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be tracked in QueueCapacities. This patch is targeting to make capacities-by-label in CS Queues are all tracked by QueueCapacities. > Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track > capacities-by-label > > > Key: YARN-3124 > URL: https://issues.apache.org/jira/browse/YARN-3124 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > > After YARN-3098, capacities-by-label (include > used-capacity/maximum-capacity/absolute-maximum-capacity, etc.) should be > tracked in QueueCapacities. > This patch is targeting to make capacities-by-label in CS Queues are all > tracked by QueueCapacities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)