[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[
https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359343#comment-17359343
]
Qi Zhu commented on YARN-10807:
---
Thanks [~bteke] for patch and [~gandras] for review.
Committed to trunk.
> Parents node labels are incorrectly added to child queues in weight mode
> -
>
> Key: YARN-10807
> URL: https://issues.apache.org/jira/browse/YARN-10807
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10807.001.patch, YARN-10807.002.patch
>
>
> In ParentQueue.updateClusterResource when calculating the normalized weights
> CS will iterate through the parent's nodelabels. If the parent has a node
> label that a specific child doesn't it will incorrectly added to the child's
> node label list through the queueCapacities.setNormalizedWeights(label,
> weight) call:
> {code:java}
> // Normalize weight of children
> if (getCapacityConfigurationTypeForQueues(childQueues)
> == QueueCapacityType.WEIGHT) {
> for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
> float sumOfWeight = 0;
> for (CSQueue queue : childQueues) {
> float weight = Math.max(0,
> queue.getQueueCapacities().getWeight(nodeLabel));
> sumOfWeight += weight;
> }
> // When sum of weight == 0, skip setting normalized_weight (so
> // normalized weight will be 0).
> if (Math.abs(sumOfWeight) > 1e-6) {
> for (CSQueue queue : childQueues) {
> queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
> queue.getQueueCapacities().getWeight(nodeLabel) /
> sumOfWeight);
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[
https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359167#comment-17359167
]
Andras Gyori commented on YARN-10807:
-
Thanks [~bteke] for the patch. I have no addition to this, looks good to me.
> Parents node labels are incorrectly added to child queues in weight mode
> -
>
> Key: YARN-10807
> URL: https://issues.apache.org/jira/browse/YARN-10807
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10807.001.patch, YARN-10807.002.patch
>
>
> In ParentQueue.updateClusterResource when calculating the normalized weights
> CS will iterate through the parent's nodelabels. If the parent has a node
> label that a specific child doesn't it will incorrectly added to the child's
> node label list through the queueCapacities.setNormalizedWeights(label,
> weight) call:
> {code:java}
> // Normalize weight of children
> if (getCapacityConfigurationTypeForQueues(childQueues)
> == QueueCapacityType.WEIGHT) {
> for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
> float sumOfWeight = 0;
> for (CSQueue queue : childQueues) {
> float weight = Math.max(0,
> queue.getQueueCapacities().getWeight(nodeLabel));
> sumOfWeight += weight;
> }
> // When sum of weight == 0, skip setting normalized_weight (so
> // normalized weight will be 0).
> if (Math.abs(sumOfWeight) > 1e-6) {
> for (CSQueue queue : childQueues) {
> queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
> queue.getQueueCapacities().getWeight(nodeLabel) /
> sumOfWeight);
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[
https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358980#comment-17358980
]
Qi Zhu commented on YARN-10807:
---
Thanks [~bteke] for update.
The patch LGTM.
> Parents node labels are incorrectly added to child queues in weight mode
> -
>
> Key: YARN-10807
> URL: https://issues.apache.org/jira/browse/YARN-10807
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10807.001.patch, YARN-10807.002.patch
>
>
> In ParentQueue.updateClusterResource when calculating the normalized weights
> CS will iterate through the parent's nodelabels. If the parent has a node
> label that a specific child doesn't it will incorrectly added to the child's
> node label list through the queueCapacities.setNormalizedWeights(label,
> weight) call:
> {code:java}
> // Normalize weight of children
> if (getCapacityConfigurationTypeForQueues(childQueues)
> == QueueCapacityType.WEIGHT) {
> for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
> float sumOfWeight = 0;
> for (CSQueue queue : childQueues) {
> float weight = Math.max(0,
> queue.getQueueCapacities().getWeight(nodeLabel));
> sumOfWeight += weight;
> }
> // When sum of weight == 0, skip setting normalized_weight (so
> // normalized weight will be 0).
> if (Math.abs(sumOfWeight) > 1e-6) {
> for (CSQueue queue : childQueues) {
> queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
> queue.getQueueCapacities().getWeight(nodeLabel) /
> sumOfWeight);
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[
https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358799#comment-17358799
]
Hadoop QA commented on YARN-10807:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
8s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 3s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m
14s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 54 unchanged - 1 fixed = 54 total (was 55) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
|
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[
https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358682#comment-17358682
]
Benjamin Teke commented on YARN-10807:
--
[~zhuqi] Thanks for the review. I agree with the suggestion, added the check
there as well. I also extended the UTs to ensure this won't be reintroduced
later.
> Parents node labels are incorrectly added to child queues in weight mode
> -
>
> Key: YARN-10807
> URL: https://issues.apache.org/jira/browse/YARN-10807
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10807.001.patch, YARN-10807.002.patch
>
>
> In ParentQueue.updateClusterResource when calculating the normalized weights
> CS will iterate through the parent's nodelabels. If the parent has a node
> label that a specific child doesn't it will incorrectly added to the child's
> node label list through the queueCapacities.setNormalizedWeights(label,
> weight) call:
> {code:java}
> // Normalize weight of children
> if (getCapacityConfigurationTypeForQueues(childQueues)
> == QueueCapacityType.WEIGHT) {
> for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
> float sumOfWeight = 0;
> for (CSQueue queue : childQueues) {
> float weight = Math.max(0,
> queue.getQueueCapacities().getWeight(nodeLabel));
> sumOfWeight += weight;
> }
> // When sum of weight == 0, skip setting normalized_weight (so
> // normalized weight will be 0).
> if (Math.abs(sumOfWeight) > 1e-6) {
> for (CSQueue queue : childQueues) {
> queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
> queue.getQueueCapacities().getWeight(nodeLabel) /
> sumOfWeight);
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[
https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358486#comment-17358486
]
Qi Zhu commented on YARN-10807:
---
Thanks [~bteke] for this work.
If we can skip the not existed also in sum logic, even thought we will sum 0
when it not exisit:
{code:java}
for (CSQueue queue : childQueues) {
float weight = Math.max(0,
queue.getQueueCapacities().getWeight(nodeLabel));
sumOfWeight += weight;
}
{code}
Other things LGTM.
Thanks.
> Parents node labels are incorrectly added to child queues in weight mode
> -
>
> Key: YARN-10807
> URL: https://issues.apache.org/jira/browse/YARN-10807
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10807.001.patch
>
>
> In ParentQueue.updateClusterResource when calculating the normalized weights
> CS will iterate through the parent's nodelabels. If the parent has a node
> label that a specific child doesn't it will incorrectly added to the child's
> node label list through the queueCapacities.setNormalizedWeights(label,
> weight) call:
> {code:java}
> // Normalize weight of children
> if (getCapacityConfigurationTypeForQueues(childQueues)
> == QueueCapacityType.WEIGHT) {
> for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
> float sumOfWeight = 0;
> for (CSQueue queue : childQueues) {
> float weight = Math.max(0,
> queue.getQueueCapacities().getWeight(nodeLabel));
> sumOfWeight += weight;
> }
> // When sum of weight == 0, skip setting normalized_weight (so
> // normalized weight will be 0).
> if (Math.abs(sumOfWeight) > 1e-6) {
> for (CSQueue queue : childQueues) {
> queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
> queue.getQueueCapacities().getWeight(nodeLabel) /
> sumOfWeight);
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[
https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357525#comment-17357525
]
Hadoop QA commented on YARN-10807:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
32s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 53s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m
6s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 55s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
