[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247504#comment-17247504
]
Eric Payne commented on YARN-9785:
--
I backported this to 2.10.
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Fix For: 3.3.0, 3.2.1, 3.1.3, 2.10.2
>
> Attachments: YARN-9785-001.patch, YARN-9785-branch-3.1.001.patch,
> YARN-9785.002.patch, YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246750#comment-17246750
]
Eric Payne commented on YARN-9785:
--
I would like to backport this to 2.10. It comes back cleanly.
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9785-001.patch, YARN-9785-branch-3.1.001.patch,
> YARN-9785.002.patch, YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921258#comment-16921258
]
Hudson commented on YARN-9785:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17222 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17222/])
YARN-9785. Fix DominantResourceCalculator when one resource is zero. (ztang:
rev bb26514ba9e8a038749c4acd592a47d5cd5bf92a)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceCalculator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921207#comment-16921207
]
Zhankun Tang commented on YARN-9785:
[~bibinchundatt], this has been committed to trunk and branch-3.2. But it
doesn't apply to branch-3.1. Could you please update the patch for branch-3.1?
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921125#comment-16921125
]
Zhankun Tang commented on YARN-9785:
+1 as well. Will commit this soon.
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921103#comment-16921103
]
Sunil Govindan commented on YARN-9785:
--
+1 from me as well. Lets get this in.
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921081#comment-16921081
]
Wangda Tan commented on YARN-9785:
--
+1 to the latest patch.
[~sunilg] do you want to take another look?
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919505#comment-16919505
]
Hadoop QA commented on YARN-9785:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new +
5 unchanged - 0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
57s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9785 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978951/YARN-9785.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9c974fc9ede1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 96f7dc1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24693/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24693/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919328#comment-16919328
]
Bibin A Chundatt commented on YARN-9785:
[~leftnoteasy] Added a testcase to verify the zero resource case
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.003.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919264#comment-16919264
]
Wangda Tan commented on YARN-9785:
--
Can we add tests to make sure no regression after this patch? And apart from
that, I think we can get isInvalidDivisor entirely, maybe we can do it in a
separate patch as this Jira blocks two releases.
Thanks,
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919250#comment-16919250
]
Rohith Sharma K S commented on YARN-9785:
-
[~sunilg]/[~leftnoteasy] Does this patch good to go?
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918493#comment-16918493
]
Hadoop QA commented on YARN-9785:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new +
1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
1s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9785 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978871/YARN-9785.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bd61fdf7cd45 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c749f62 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24672/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24672/testReport/ |
| Max. process+thread count | 336 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918445#comment-16918445
]
Sunil Govindan commented on YARN-9785:
--
Thanks [~bibinchundatt]
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918444#comment-16918444
]
Bibin A Chundatt commented on YARN-9785:
Renamed wip patch to 002.patch and uploaded again.
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch,
> YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918380#comment-16918380
]
Bilwa S T commented on YARN-9785:
-
Hi [~leftnoteasy]
I had added *resourceCalculator.fitsIn* to make sure AmIfStarted is less than
AMLimit because lessThanOrEqual will fail in a case where lhs.memory <
rhs.memory && lhs.vcores > rhs.vcores . I think this part can be handled in
Jira YARN-9797 .
[~sunilg] WIP patch looks good. It handles if any resource is 0. We can go
ahead with WIP patch.
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918357#comment-16918357
]
Wangda Tan commented on YARN-9785:
--
Thanks [~BilwaST] for the patch and everybody for discussing.
I'm doubt if this is a right fix or not:
{code:java}
public static boolean lessThanOrEqual(
ResourceCalculator resourceCalculator,
Resource clusterResource,
Resource lhs, Resource rhs) {
return resourceCalculator.fitsIn(lhs, rhs)
&& resourceCalculator.compare(clusterResource, lhs, rhs) <= 0;
} {code}
If lhs fits in rhs, the check after && is not need correct? And when we check
less than or equal, we check dominante value of the two resources, I don't
understand why we check fitsIn here.
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918310#comment-16918310
]
Sunil Govindan commented on YARN-9785:
--
Thanks [~bibinchundatt] for helping to clarify the issues. As far as we
discussed, we are dealing with TWO separate issues here.
# When a new resource type is added and no resources are configured for that
resource type, then we have 0 in the resource entry. This causes the comparison
issues. With the wip patch, we can resolve this pblm.
# However as you pointed out in the 3rd case in above comment , Dominant
resource calculator will kick in as CPU is more on RHS and Memory is more on
LHS. But in ratio level, RHS has upper hand and CPU will become dominant here.
This literally messes up the comparison of AM limi in Leaf Queue. As you
mentioned, this can be fixed by changing to use fitsIn instead of
lessThanOrEquals method.
Could we create a new Jira to track issue #2 mentioned here for AMLimit and
this current issue can track the GPU resource 0 issue alone.
Thanks
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero
[
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918308#comment-16918308
]
Bibin A Chundatt commented on YARN-9785:
[~sunilg]/[~rohithsharma]
Raise YARN-9797 to fix the AM activation issue
> Fix DominantResourceCalculator when one resource is zero
>
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
> yarn.resource-types
> yarn.io/gpu
>
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications
> get activated even after limit is reached
> !queue.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
