[jira] [Commented] (YARN-10813) Root queue capacity is not set when using node labels
[
https://issues.apache.org/jira/browse/YARN-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363516#comment-17363516
]
Andras Gyori commented on YARN-10813:
-
I think this is better be addressed after YARN-10780 is merged.
> Root queue capacity is not set when using node labels
> -
>
> Key: YARN-10813
> URL: https://issues.apache.org/jira/browse/YARN-10813
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10813.001.patch
>
>
> CapacitySchedulerConfiguration#getNonLabeledQueueCapacity handles root in the
> following way:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource will be parsed and
> // updated. Once nodes are added/removed in cluster, capacity in
> // percentage will also be re-calculated.
> return queue.equals("root") ? 100.0f : 0f;
> }
> {code}
> CapacitySchedulerConfiguration#internalGetLabeledQueueCapacity on the other
> hand does not take root queue into consideration:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource, and weight will be
> parsed
> // and updated separately. Once nodes are added/removed in cluster,
> // capacity is percentage will also be re-calculated.
> return defaultValue;
> }
> float capacity = getFloat(capacityPropertyName, defaultValue);
> {code}
> Due to this, labeled root capacity is 0, which is not set in in
> AbstractCSQueue#derivedCapacityFromAbsoluteConfigurations, because root is
> never in Absolute mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10813) Root queue capacity is not set when using node labels
[
https://issues.apache.org/jira/browse/YARN-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363475#comment-17363475
]
Andras Gyori commented on YARN-10813:
-
Thank you [~snemeth] for your suggestions. It seems that this fix is just the
tip of the iceberg. The problem is that:
* A user can configure a node label for a queue
* But without setting the capacity 100 for ROOT to the same label manually,
eg. yarn.scheduler.capacity.root.accessible-node-labels.test_label.capacity
100, the absoluteCapacity will be 0 all the time.
* This issue stands for both percentage and absolute resource as well, and the
node labels for absolute resources were not even tested (see
TestAbsoluteResourceConfiguration).
A solution to this problem could be for capacity to default to 100 for every
undefined node label.
> Root queue capacity is not set when using node labels
> -
>
> Key: YARN-10813
> URL: https://issues.apache.org/jira/browse/YARN-10813
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10813.001.patch
>
>
> CapacitySchedulerConfiguration#getNonLabeledQueueCapacity handles root in the
> following way:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource will be parsed and
> // updated. Once nodes are added/removed in cluster, capacity in
> // percentage will also be re-calculated.
> return queue.equals("root") ? 100.0f : 0f;
> }
> {code}
> CapacitySchedulerConfiguration#internalGetLabeledQueueCapacity on the other
> hand does not take root queue into consideration:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource, and weight will be
> parsed
> // and updated separately. Once nodes are added/removed in cluster,
> // capacity is percentage will also be re-calculated.
> return defaultValue;
> }
> float capacity = getFloat(capacityPropertyName, defaultValue);
> {code}
> Due to this, labeled root capacity is 0, which is not set in in
> AbstractCSQueue#derivedCapacityFromAbsoluteConfigurations, because root is
> never in Absolute mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10813) Root queue capacity is not set when using node labels
[
https://issues.apache.org/jira/browse/YARN-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363185#comment-17363185
]
Szilard Nemeth commented on YARN-10813:
---
Thanks [~gandras] for reporting this,
This is quite a trivial fix, but it's good that you have spotted it.
Some questions / observations:
1. How come our tests didn't catch this? Is it easy to add a unit test to cover
the fixed scenario if it isn't any?
2. I would have been baffled to realize if we don't have a common constant for
the queue name "root" anywhere. The thing is, we have many constants, just
search for "root" from package:
org/apache/hadoop/yarn/server/resourcemanager/scheduler
I know it's not strongly related to this, but could you please report a
follow-up to clean up those? I just don't want to increase the number of
occurrences of "root" in production code anymore.
Thanks.
> Root queue capacity is not set when using node labels
> -
>
> Key: YARN-10813
> URL: https://issues.apache.org/jira/browse/YARN-10813
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10813.001.patch
>
>
> CapacitySchedulerConfiguration#getNonLabeledQueueCapacity handles root in the
> following way:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource will be parsed and
> // updated. Once nodes are added/removed in cluster, capacity in
> // percentage will also be re-calculated.
> return queue.equals("root") ? 100.0f : 0f;
> }
> {code}
> CapacitySchedulerConfiguration#internalGetLabeledQueueCapacity on the other
> hand does not take root queue into consideration:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource, and weight will be
> parsed
> // and updated separately. Once nodes are added/removed in cluster,
> // capacity is percentage will also be re-calculated.
> return defaultValue;
> }
> float capacity = getFloat(capacityPropertyName, defaultValue);
> {code}
> Due to this, labeled root capacity is 0, which is not set in in
> AbstractCSQueue#derivedCapacityFromAbsoluteConfigurations, because root is
> never in Absolute mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10813) Root queue capacity is not set when using node labels
[
https://issues.apache.org/jira/browse/YARN-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362825#comment-17362825
]
Benjamin Teke commented on YARN-10813:
--
Thanks [~gandras] for the patch, this indeed seems to be a bug. LGTM
(non-binding).
> Root queue capacity is not set when using node labels
> -
>
> Key: YARN-10813
> URL: https://issues.apache.org/jira/browse/YARN-10813
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10813.001.patch
>
>
> CapacitySchedulerConfiguration#getNonLabeledQueueCapacity handles root in the
> following way:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource will be parsed and
> // updated. Once nodes are added/removed in cluster, capacity in
> // percentage will also be re-calculated.
> return queue.equals("root") ? 100.0f : 0f;
> }
> {code}
> CapacitySchedulerConfiguration#internalGetLabeledQueueCapacity on the other
> hand does not take root queue into consideration:
> {code:java}
> if (absoluteResourceConfigured || configuredWeightAsCapacity(
> configuredCapacity)) {
> // Return capacity in percentage as 0 for non-root queues and 100 for
> // root.From AbstractCSQueue, absolute resource, and weight will be
> parsed
> // and updated separately. Once nodes are added/removed in cluster,
> // capacity is percentage will also be re-calculated.
> return defaultValue;
> }
> float capacity = getFloat(capacityPropertyName, defaultValue);
> {code}
> Due to this, labeled root capacity is 0, which is not set in in
> AbstractCSQueue#derivedCapacityFromAbsoluteConfigurations, because root is
> never in Absolute mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10813) Root queue capacity is not set when using node labels
[
https://issues.apache.org/jira/browse/YARN-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360140#comment-17360140
]
Hadoop QA commented on YARN-10813:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
57s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
7s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 48s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 23m
37s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
14s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 24s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
|
