[
https://issues.apache.org/jira/browse/YARN-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215015#comment-15215015
]
Eric Payne commented on YARN-4751:
----------------------------------
Thank you very much, [~leftnoteasy]!
{quote}
1) The ultimate solution seems to be YARN-3362. Have you evaluated how hard to
back port it?
{quote}
My intent with this JIRA was simply to provide accurate visual feedback on the
Capacity Scheduler Application Queues page for resources used by labeled
queues. I will look deeper into what it would take to backport YARN-3362. My
concern is that it would include more risk and features than we really want to
backport into 2.7.
{quote}
2) If you don't want to backport YARN-3362. IIUC, the computation of
total-used-capacity-considers-all-labels seems wrong:
In your patch it is Σ(queue.label.used_capacity), actually it should be
Σ(queue.label.used_resource) / S(root.label.total_resource)
{quote}
I think you are talking about {{getAllUsedCapacityConsideringLabels}}. IIUC, in
this patch, used capacity is calculated correctly by
{{CSQueueUtils#updateQueueStatistics}} when it is called by {{LeafQueue}} to
allocate resources. {{updateQueueStatistics}} calls
{{AbstractCSQueue#getAllUsedResourcesConsideringLabels}} to retrieve all
resources consumed by the queue, including labeled resources. It then divides
the consumed resources by the queue's limit:
{code}
public static void updateQueueStatistics(
...
Resource usedResources = childQueue.getAllUsedResourcesConsideringLabels();
...
usedCapacity =
Resources.equals(queueLimit, Resources.none()) ? 0 :
Resources.divide(calculator, clusterResource,
usedResources, queueLimit);
...
childQueue.setUsedCapacity(usedCapacity);
{code}
Even still, this patch is, perhaps, not ideal. When used capacity, absolute
used capacity, and available resources are set, these metrics are associated
with {{NO_LABEL}}. So, although the metrics reflect all used resources for a
queue (even at parent levels), they are not set per label but just gathered in
with {{NO_LABEL}}. In order to associate the resources with a label, I think
the interface to {{CSQueueUtils#updateQueueStatistics}} may need to change.
Let me get a more precise view of what it would take to backport YARN-3362.
Thanks again.
> In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
> -----------------------------------------------------------------------
>
> Key: YARN-4751
> URL: https://issues.apache.org/jira/browse/YARN-4751
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
> Affects Versions: 2.7.3
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: 2.7 CS UI No BarGraph.jpg,
> YARH-4752-branch-2.7.001.patch, YARH-4752-branch-2.7.002.patch
>
>
> In 2.6 and 2.7, the capacity scheduler UI does not have the queue graphs
> separated by partition. When applications are running on a labeled queue, no
> color is shown in the bar graph, and several of the "Used" metrics are zero.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
