[
https://issues.apache.org/jira/browse/YARN-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-4751:
-----------------------------
Attachment: YARN-4751-branch-2.7.003.patch
Thank you, [~leftnoteasy] and [~sunilg], for your reviews and helpful insights.
I backported YARN-3362 to 2.7, but it did not provide all of the correct
statistics for labeled queues all by itself.
- Used Capacity was not calculated correctly for node labels because
{{QueueCapacities#setUsedCapacity}} does not have a label-specific interface
that could set the capacity per partition. In order to get the updated changes
to {{CSQueueUtils#updateQueueStatistics}} that set used capacity per label, I
would need to backport YARN-2696. I did not do that since it also had other
dependencies and was more invasive than I felt was needed.
-- Instead, I added a {{getUsedCapacity}} method to {{AbstractCSQueue}} that
takes a label and returns the computed used capacity for the queue for that
label.
- The {{RMNodeLabel}} class is named {{NodeLabel}} in 2.7.
- The {{CapacitySchedulerInfo}} constructor doesn't have the
{{CapacityScheduler}} parameter in 2.7, so I left it out (as you suggested
previously).
- The YARN-3362 patch has a dependency on
{{AbstractCSQueue#accessibleToPartition}} which was not in 2.7. Rather than
backport YARN-3361 as well, I just recrated {{accessibleToPartition}} in this
patch (as you suggested previously).
- One of the changes for {{CapacitySchedulerInfo#getQueues}} in YARN-3362 added
a check to skip non-accessible queues. In 2.7, there are 2 loops that iterate
over the queues whereas in 2.8, there is only 1. I put the check in the first
loop because the first loop is filtering the queues to be processed by the
second loop
- I modified {{AbstractCSQueue#allocateResource}} and {{releaseResource}} to
intersect with all labels in the passed {{nodeLabels}} set if the accessible
node labels included the {{ANY}} selection. Without this, queue usage for root
would not be preserved per-partiton and the root queue's bar graph would not
display any usage at all for specific partitions.
- Modified {{LeafQueue#getUser}} to sum all labeled and unlabeld used resources
for a user on a queue. Without this, the {{Used Resource}} in the {{Active
Users Info}} section would be 0.
> In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
> -----------------------------------------------------------------------
>
> Key: YARN-4751
> URL: https://issues.apache.org/jira/browse/YARN-4751
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
> Affects Versions: 2.7.3
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: 2.7 CS UI No BarGraph.jpg,
> YARH-4752-branch-2.7.001.patch, YARH-4752-branch-2.7.002.patch,
> YARN-4751-branch-2.7.003.patch
>
>
> In 2.6 and 2.7, the capacity scheduler UI does not have the queue graphs
> separated by partition. When applications are running on a labeled queue, no
> color is shown in the bar graph, and several of the "Used" metrics are zero.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
