[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[ https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886781#comment-16886781 ] Hari Sekhon commented on YARN-3638: --- I think it would be better if it was to the side of the queue bar as that's more intuitive, as long as both the % of configured and % of total cluster are clearly labelled. > Yarn Resource Manager Scheduler page - show percentage of total cluster that > each queue is using > > > Key: YARN-3638 > URL: https://issues.apache.org/jira/browse/YARN-3638 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.6.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon >Assignee: Kuhu Shukla >Priority: Minor > Attachments: Capacity Scheduler Page Shows Percentages of Cluster.png > > > Request to show % of total cluster resources each queue is currently > consuming for jobs on the Yarn Resource Manager Scheduler page. > Currently the Yarn Resource Manager Scheduler page shows the % of total used > for root queue and the % of each given queue's configured capacity that is > used (often showing say 150% if the max capacity is greater than configured > capacity to allow bursting where there are free resources). This is fine, but > it would be good to additionally show the % of total cluster that each given > queue is consuming and not just the % of that queue's configured capacity. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[ https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886391#comment-16886391 ] Eric Payne commented on YARN-3638: -- Hi [~harisekhon], bq. You could also extend this idea show the % of total cluster capacity that each job is consuming too. I attached a screenshot of the Capacity Scheduler UI page so that we could all be on the same page. As you can see, the capacity scheduler page shows the absolute used capacity for each leaf queue, as well as the absolute capacity being used by each app. bq. I'd like "Absolute Used Capacity" to be shown to the right of the horizontal bars. You are correct in that the information is not displayed to the right along with the used capacity. This is the one piece that is missing. Is this feature still given that all of the information is there (albeit not quite as convenient)? > Yarn Resource Manager Scheduler page - show percentage of total cluster that > each queue is using > > > Key: YARN-3638 > URL: https://issues.apache.org/jira/browse/YARN-3638 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.6.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon >Assignee: Kuhu Shukla >Priority: Minor > Attachments: Capacity Scheduler Page Shows Percentages of Cluster.png > > > Request to show % of total cluster resources each queue is currently > consuming for jobs on the Yarn Resource Manager Scheduler page. > Currently the Yarn Resource Manager Scheduler page shows the % of total used > for root queue and the % of each given queue's configured capacity that is > used (often showing say 150% if the max capacity is greater than configured > capacity to allow bursting where there are free resources). This is fine, but > it would be good to additionally show the % of total cluster that each given > queue is consuming and not just the % of that queue's configured capacity. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[ https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541885#comment-14541885 ] Jason Lowe commented on YARN-3638: -- Isn't this the Absolute Used Capacity metric that is shown for each leaf queue? Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using Key: YARN-3638 URL: https://issues.apache.org/jira/browse/YARN-3638 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler, yarn Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Priority: Minor Request to show % of total cluster resources each queue is currently consuming for jobs on the Yarn Resource Manager Scheduler page. Currently the Yarn Resource Manager Scheduler page shows the % of total used for root queue and the % of each given queue's configured capacity that is used (often showing say 150% if the max capacity is greater than configured capacity to allow bursting where there are free resources). This is fine, but it would be good to additionally show the % of total cluster that each given queue is consuming and not just the % of that queue's configured capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[ https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541950#comment-14541950 ] Hari Sekhon commented on YARN-3638: --- [~jlowe] yes I believe it is % of absolute capacity that is shown, which is useful to seeing how much you're bursting over. It would be nice if RM would also show the % of the total cluster's capacity that the leaf queue was consuming. You could also extend this idea show the % of total cluster capacity that each job is consuming too. Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using Key: YARN-3638 URL: https://issues.apache.org/jira/browse/YARN-3638 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler, yarn Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Priority: Minor Request to show % of total cluster resources each queue is currently consuming for jobs on the Yarn Resource Manager Scheduler page. Currently the Yarn Resource Manager Scheduler page shows the % of total used for root queue and the % of each given queue's configured capacity that is used (often showing say 150% if the max capacity is greater than configured capacity to allow bursting where there are free resources). This is fine, but it would be good to additionally show the % of total cluster that each given queue is consuming and not just the % of that queue's configured capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[
https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541969#comment-14541969
]
Jason Lowe commented on YARN-3638:
--
bq. I believe it is % of absolute capacity that is shown
No, the Absolute Used Capacity field is the amount of total cluster capacity
being used by this queue. From the 2.6 code in
CSQueueUtils.updateQueueStatistics:
{code}
absoluteUsedCapacity =
Resources.divide(calculator, clusterResource,
usedResources, clusterResource);
{code}
Yarn Resource Manager Scheduler page - show percentage of total cluster that
each queue is using
Key: YARN-3638
URL: https://issues.apache.org/jira/browse/YARN-3638
Project: Hadoop YARN
Issue Type: Improvement
Components: capacityscheduler, resourcemanager, scheduler, yarn
Affects Versions: 2.6.0
Environment: HDP 2.2
Reporter: Hari Sekhon
Priority: Minor
Request to show % of total cluster resources each queue is currently
consuming for jobs on the Yarn Resource Manager Scheduler page.
Currently the Yarn Resource Manager Scheduler page shows the % of total used
for root queue and the % of each given queue's configured capacity that is
used (often showing say 150% if the max capacity is greater than configured
capacity to allow bursting where there are free resources). This is fine, but
it would be good to additionally show the % of total cluster that each given
queue is consuming and not just the % of that queue's configured capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[ https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542563#comment-14542563 ] Wangda Tan commented on YARN-3638: -- I think this is useful. Maybe one possible way is add a switch in RM scheduler UI to change used-capacity and absolute-used-capacity showing in queue bars. Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using Key: YARN-3638 URL: https://issues.apache.org/jira/browse/YARN-3638 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler, yarn Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Priority: Minor Request to show % of total cluster resources each queue is currently consuming for jobs on the Yarn Resource Manager Scheduler page. Currently the Yarn Resource Manager Scheduler page shows the % of total used for root queue and the % of each given queue's configured capacity that is used (often showing say 150% if the max capacity is greater than configured capacity to allow bursting where there are free resources). This is fine, but it would be good to additionally show the % of total cluster that each given queue is consuming and not just the % of that queue's configured capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[ https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542587#comment-14542587 ] Jason Lowe commented on YARN-3638: -- Ah, got it. I originally thought the complaint was the user cannot find that information anywhere on the scheduler page, but it is there for all the leaf queues (in the details accessed by the twisty to the left of the queue). I'm OK with having the percentage to the right of the queue bar being an absolute used percentage as long as it's clear that's what it is representing. Otherwise I can already see the JIRA stating the percentage shown to the right of the queue doesn't match the relative length of the bars to their capacity. ;-) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using Key: YARN-3638 URL: https://issues.apache.org/jira/browse/YARN-3638 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler, yarn Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Priority: Minor Request to show % of total cluster resources each queue is currently consuming for jobs on the Yarn Resource Manager Scheduler page. Currently the Yarn Resource Manager Scheduler page shows the % of total used for root queue and the % of each given queue's configured capacity that is used (often showing say 150% if the max capacity is greater than configured capacity to allow bursting where there are free resources). This is fine, but it would be good to additionally show the % of total cluster that each given queue is consuming and not just the % of that queue's configured capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3638) Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using
[ https://issues.apache.org/jira/browse/YARN-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542171#comment-14542171 ] Hari Sekhon commented on YARN-3638: --- Sorry the wording of these fields is sub-optimal, I meant to say the Used Capacity of the Absolute Capacity [configured] is what's shown, and I'd like Absolute Used Capacity to be shown to the right of the horizontal bars. It would make more intuitive sense to show that 50% of the total cluster is being used by a given queue than that the currently shown Used Capacity of 150% of what that queue had configured as dedicated capacity. In any event, both can be shown so you can see at a glance what's being used out of the total cluster between the queues, as well as how much over the dedicated capacity the queue is consuming (which is what's currently shown). Yarn Resource Manager Scheduler page - show percentage of total cluster that each queue is using Key: YARN-3638 URL: https://issues.apache.org/jira/browse/YARN-3638 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler, yarn Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Priority: Minor Request to show % of total cluster resources each queue is currently consuming for jobs on the Yarn Resource Manager Scheduler page. Currently the Yarn Resource Manager Scheduler page shows the % of total used for root queue and the % of each given queue's configured capacity that is used (often showing say 150% if the max capacity is greater than configured capacity to allow bursting where there are free resources). This is fine, but it would be good to additionally show the % of total cluster that each given queue is consuming and not just the % of that queue's configured capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
