[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[
https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883493#comment-16883493
]
Hadoop QA commented on YARN-9668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} branch-2 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} branch-2 passed with JDK v1.8.0_212 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
37s{color} | {color:red} hadoop-yarn-server in the patch failed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s{color}
| {color:red} hadoop-yarn-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
29s{color} | {color:red} hadoop-yarn-server in the patch failed with JDK
v1.8.0_212. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s{color}
| {color:red} hadoop-yarn-server in the patch failed with JDK v1.8.0_212.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m
45s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:da675796017 |
| JIRA Issue | YARN-9668 |
| JIRA Patch URL |
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[
https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883447#comment-16883447
]
Hadoop QA commented on YARN-9668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
21s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
15s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 18s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 33s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}195m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | YARN-9668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974451/YARN-9668-branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3672847bb4d9 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 3e3bbb7 |
| maven
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[
https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883439#comment-16883439
]
Hadoop QA commented on YARN-9668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
48s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
11s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 5s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:63396be |
| JIRA Issue | YARN-9668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974451/YARN-9668-branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux cf9d6ac90864 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 3e3bbb7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883389#comment-16883389 ] Jonathan Hung commented on YARN-9668: - Yes. Uploaded branch-2 patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668-branch-2.001.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9360) Do not expose innards of QueueMetrics object into FSLeafQueue#computeMaxAMResource
[
https://issues.apache.org/jira/browse/YARN-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883382#comment-16883382
]
Hadoop QA commented on YARN-9360:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m
7s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9360 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974445/YARN-9360.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9259997f7057 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6a3433b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24380/testReport/ |
| Max. process+thread count | 897 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24380/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883374#comment-16883374 ] Haibo Chen commented on YARN-9668: -- [~jhung] Do you intend to fix this for branch-2 as well?The branch-3.2 does not apply cleanly to branch-2,. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883358#comment-16883358 ] Jonathan Hung commented on YARN-9668: - Uploaded branch-3.2 version. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9337) GPU auto-discovery script runs even when the resource is given by hand
[
https://issues.apache.org/jira/browse/YARN-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883356#comment-16883356
]
Hadoop QA commented on YARN-9337:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 41m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
6s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9337 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1297/YARN-9337.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fbd0ea13f310 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6a3433b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24378/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24378/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9688-branch-3.2.001.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: (was: YARN-9688-branch-3.2.001.patch) > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668-branch-3.2.001.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883341#comment-16883341 ] Hudson commented on YARN-9668: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16895 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16895/]) YARN-9668. UGI conf doesn't read user overridden configurations on RM (haibochen: rev 9b54dd71863af58c2ef5f2194f5097626b8ef18a) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManager.java > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[
https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1688#comment-1688
]
Hadoop QA commented on YARN-9356:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
38s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9356 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962965/YARN-9356.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e18e6906f031 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6a3433b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24377/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24377/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add more tests to ratio method in TestResourceCalculator
> -
>
> Key:
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883326#comment-16883326 ] Haibo Chen commented on YARN-9668: -- +1 on the latest patch. Committing it shortly. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[
https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883315#comment-16883315
]
Hadoop QA commented on YARN-9140:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new +
3 unchanged - 6 fixed = 4 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
39s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9140 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974382/YARN-9140.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux ef223eecdcd0 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6a3433b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24376/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24376/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[
https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883310#comment-16883310
]
Hadoop QA commented on YARN-9217:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-9217 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957894/YARN-9217.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24379/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Nodemanager will fail to start if GPU is misconfigured on the node or GPU
> drivers missing
> -
>
> Key: YARN-9217
> URL: https://issues.apache.org/jira/browse/YARN-9217
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9217.001.patch, YARN-9217.002.patch,
> YARN-9217.003.patch, YARN-9217.004.patch
>
>
> Nodemanager will not start
> 1. If Autodiscovery is enabled:
> * If nvidia-smi path is misconfigured or the file does not exist.
> * There is 0 GPU found
> * If the file exists but it is not pointing to an nvidia-smi
> * if the binary is ok but there is an IOException
> 2. If the manually configured GPU devices are misconfigured
> * Any index:minor number format failure will cause a problem
> * 0 configured device will cause a problem
> * NumberFormatException is not handled
> It would be a better option to add warnings about the configuration, set 0
> available GPUs and let the node work and run non-gpu jobs.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[ https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-9217: Assignee: Peter Bacsko (was: Antal Bálint Steinbach) > Nodemanager will fail to start if GPU is misconfigured on the node or GPU > drivers missing > - > > Key: YARN-9217 > URL: https://issues.apache.org/jira/browse/YARN-9217 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.0 >Reporter: Antal Bálint Steinbach >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9217.001.patch, YARN-9217.002.patch, > YARN-9217.003.patch, YARN-9217.004.patch > > > Nodemanager will not start > 1. If Autodiscovery is enabled: > * If nvidia-smi path is misconfigured or the file does not exist. > * There is 0 GPU found > * If the file exists but it is not pointing to an nvidia-smi > * if the binary is ok but there is an IOException > 2. If the manually configured GPU devices are misconfigured > * Any index:minor number format failure will cause a problem > * 0 configured device will cause a problem > * NumberFormatException is not handled > It would be a better option to add warnings about the configuration, set 0 > available GPUs and let the node work and run non-gpu jobs. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[ https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883303#comment-16883303 ] Szilard Nemeth commented on YARN-9217: -- Hi [~pbacsko]! Please take over this jira, fix the comments and I will make sure to commit this soon. > Nodemanager will fail to start if GPU is misconfigured on the node or GPU > drivers missing > - > > Key: YARN-9217 > URL: https://issues.apache.org/jira/browse/YARN-9217 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.0 >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Major > Attachments: YARN-9217.001.patch, YARN-9217.002.patch, > YARN-9217.003.patch, YARN-9217.004.patch > > > Nodemanager will not start > 1. If Autodiscovery is enabled: > * If nvidia-smi path is misconfigured or the file does not exist. > * There is 0 GPU found > * If the file exists but it is not pointing to an nvidia-smi > * if the binary is ok but there is an IOException > 2. If the manually configured GPU devices are misconfigured > * Any index:minor number format failure will cause a problem > * 0 configured device will cause a problem > * NumberFormatException is not handled > It would be a better option to add warnings about the configuration, set 0 > available GPUs and let the node work and run non-gpu jobs. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9360) Do not expose innards of QueueMetrics object into FSLeafQueue#computeMaxAMResource
[ https://issues.apache.org/jira/browse/YARN-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883292#comment-16883292 ] Szilard Nemeth commented on YARN-9360: -- As we agreed on this offline, [~pbacsko] is taking over this patch from me. > Do not expose innards of QueueMetrics object into > FSLeafQueue#computeMaxAMResource > -- > > Key: YARN-9360 > URL: https://issues.apache.org/jira/browse/YARN-9360 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9360.001.patch, YARN-9360.002.patch, > YARN-9360.003.patch, YARN-9360.003.patch > > > This is a follow-up for YARN-9323, covering required changes as discussed > with [~templedf] earlier. > After YARN-9323, > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue#computeMaxAMResource > gets the QueueMetricsForCustomResources object from > scheduler.getRootQueueMetrics(). > Instead, we should use a "fill-in" method in QueueMetrics that receives a > Resource and fills in custom resource values if they are non-zero. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9360) Do not expose innards of QueueMetrics object into FSLeafQueue#computeMaxAMResource
[ https://issues.apache.org/jira/browse/YARN-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9360: - Attachment: YARN-9360.003.patch > Do not expose innards of QueueMetrics object into > FSLeafQueue#computeMaxAMResource > -- > > Key: YARN-9360 > URL: https://issues.apache.org/jira/browse/YARN-9360 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9360.001.patch, YARN-9360.002.patch, > YARN-9360.003.patch, YARN-9360.003.patch > > > This is a follow-up for YARN-9323, covering required changes as discussed > with [~templedf] earlier. > After YARN-9323, > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue#computeMaxAMResource > gets the QueueMetricsForCustomResources object from > scheduler.getRootQueueMetrics(). > Instead, we should use a "fill-in" method in QueueMetrics that receives a > Resource and fills in custom resource values if they are non-zero. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883289#comment-16883289 ] Szilard Nemeth commented on YARN-9356: -- Talked about this offline with [~zsiegl], we agreed on that we don't need more testcases to add. > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883289#comment-16883289 ] Szilard Nemeth edited comment on YARN-9356 at 7/11/19 7:46 PM: --- Talked about this offline with [~zsiegl], we agreed on that we don't need more testcases to add. +1 (binding), committing this one soon. was (Author: snemeth): Talked about this offline with [~zsiegl], we agreed on that we don't need more testcases to add. > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9337) GPU auto-discovery script runs even when the resource is given by hand
[ https://issues.apache.org/jira/browse/YARN-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883288#comment-16883288 ] Szilard Nemeth commented on YARN-9337: -- Re-attached latest patch in order to have a fresh jenkins result. > GPU auto-discovery script runs even when the resource is given by hand > -- > > Key: YARN-9337 > URL: https://issues.apache.org/jira/browse/YARN-9337 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9337.001.patch, YARN-9337.002.patch, > YARN-9337.003.patch, YARN-9337.003.patch > > > The nvidia-smi script is called even when the gpu configs are given by hand > (so there's no need for GPU auto-discovery). > We should mitigate the call of that script, since it has no effect. (The > configs written by the user is not overwritten by the result of the > auto-discovery script.) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9337) GPU auto-discovery script runs even when the resource is given by hand
[ https://issues.apache.org/jira/browse/YARN-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9337: - Attachment: YARN-9337.003.patch > GPU auto-discovery script runs even when the resource is given by hand > -- > > Key: YARN-9337 > URL: https://issues.apache.org/jira/browse/YARN-9337 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9337.001.patch, YARN-9337.002.patch, > YARN-9337.003.patch, YARN-9337.003.patch > > > The nvidia-smi script is called even when the gpu configs are given by hand > (so there's no need for GPU auto-discovery). > We should mitigate the call of that script, since it has no effect. (The > configs written by the user is not overwritten by the result of the > auto-discovery script.) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9235) If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown
[
https://issues.apache.org/jira/browse/YARN-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883284#comment-16883284
]
Szilard Nemeth commented on YARN-9235:
--
Hi [~bsteinbach]!
If you don't mind, [~adam.antal] could take this jira over as this is an
important bug.
> If linux container executor is not set for a GPU cluster
> GpuResourceHandlerImpl is not initialized and NPE is thrown
>
>
> Key: YARN-9235
> URL: https://issues.apache.org/jira/browse/YARN-9235
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Major
> Attachments: YARN-9235.001.patch, YARN-9235.002.patch,
> YARN-9235.003.patch, YARN-9235.004.patch
>
>
> If GPU plugin is enabled for the NodeManager, it is possible to run jobs with
> GPU.
> However, if LinuxContainerExecutor is not configured, an NPE is thrown when
> calling
> {code:java}
> GpuResourcePlugin.getNMResourceInfo{code}
> Also, there are no warns in the log if GPU is misconfigured like this.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883282#comment-16883282 ] Szilard Nemeth commented on YARN-9140: -- [~pbacsko] Indeed, you are right. Here's the [link|https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-YARN-Build/24366/console] to the build, the overall result was -1, but somehow the comment is not posted here. You can check the results in the console log of jenkins. Retriggered the build, hopefully we will have the results here as well. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883228#comment-16883228 ] Szilard Nemeth commented on YARN-9625: -- Thanks [~zsiegl] for the contribution! Committed patches to trunk, branch-3.2 and branch-3.1 > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch, > YARN-9625.branch-3.1.001.patch, YARN-9625.branch-3.2.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9625: - Attachment: YARN-9625.branch-3.1.001.patch > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch, > YARN-9625.branch-3.1.001.patch, YARN-9625.branch-3.2.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883224#comment-16883224 ] Szilard Nemeth commented on YARN-9573: -- Hi [~adam.antal]! Thanks for the patches, just pushed 3.2 / 3.1 patches. Thanks [~sunilg] for the reviews! > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.branch-3.1.patch, > YARN-9573.001.branch-3.2.patch, YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9671) Improve Locality Scheduling when cluster is busy
Muhammad Samir Khan created YARN-9671: - Summary: Improve Locality Scheduling when cluster is busy Key: YARN-9671 URL: https://issues.apache.org/jira/browse/YARN-9671 Project: Hadoop YARN Issue Type: Improvement Reporter: Muhammad Samir Khan Assignee: Muhammad Samir Khan When a cluster is very busy, scheduling opportunities are few and far between. Scheduling opportunities are how an application knows when to give up looking for decent locality. It doesn't make sense to work hard waiting for locality when the odds of it coming are very small and it may actually take a very long time to actually give up. This causes the priority of queues to be violated which is the last thing we want to do when the cluster is full. * Add metrics for queue priority inversions. * Add mode to disable skipping locality when cluster is busy. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9100) Add tests for GpuResourceAllocator and do minor code cleanup
[
https://issues.apache.org/jira/browse/YARN-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883148#comment-16883148
]
Hadoop QA commented on YARN-9100:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 13 unchanged - 6 fixed = 13 total (was 19) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
6s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9100 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974378/YARN-9100-005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fbed6aa58158 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9cec023 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24375/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24375/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[
https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883149#comment-16883149
]
Hadoop QA commented on YARN-9127:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
58s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 5 unchanged - 5 fixed = 5 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
33s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 7s{color} |
{color:red} The patch has 4 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
|
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
|
|
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[
https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883095#comment-16883095
]
Hadoop QA commented on YARN-9625:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
47s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
37m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | YARN-9625 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974409/YARN-9625.branch-3.2.001.patch
|
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux 6c87c1d504e4 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / b6718c7 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24374/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 - No link to a queue on the Queues page for Fair Scheduler
> --
>
> Key: YARN-9625
> URL: https://issues.apache.org/jira/browse/YARN-9625
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png,
> YARN-9625-after-patch.png, YARN-9625.001.patch, YARN-9625.branch-3.2.001.patch
>
>
> When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab
> on the right with a link to a certain queue which provides running app
> information for the queue. But for 'Fair Scheduler' there is no such link.
> Attached screenshots for both schedulers.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9100) Add tests for GpuResourceAllocator and do minor code cleanup
[ https://issues.apache.org/jira/browse/YARN-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883072#comment-16883072 ] Peter Bacsko commented on YARN-9100: Ok, I had to retrigger the build for v5. It just failed for no obvious reason. Let's wait for it. > Add tests for GpuResourceAllocator and do minor code cleanup > > > Key: YARN-9100 > URL: https://issues.apache.org/jira/browse/YARN-9100 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9100-004.patch, YARN-9100-005.patch, > YARN-9100.001.patch, YARN-9100.002.patch, YARN-9100.003.patch > > > Add tests for GpuResourceAllocator and do minor code cleanup > - Improved log and exception messages > - Added some new debug logs > - Some methods are named like *Copy, these are returning copies of internal > data structures. The word "copy" is just a noise in their name, so they have > been renamed. Additionally, the copied data structures modified to be > immutable. > - The waiting loop in method assignGpus were decoupled into a new class, > RetryCommand. > Some more words about the new class RetryCommand: > There are some similar waiting loops in the code in: AMRMClient, > AMRMClientAsync and even in GenericTestUtils (see waitFor method). > RetryCommand could be a future replacement of these duplicated code, as it > gives a solution to this waiting loop problem in a generic way. > The only downside of the usage of RetryCommand in GpuResourceAllocator > (startGpuAssignmentLoop) is the ugly exception handling part, but that's > solely because how Java deals with checked exceptions vs. lambdas. If there's > a cleaner way to solve the exception handling, I'm open for any suggestions. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9135) NM State store ResourceMappings serialization are tested with Strings instead of real Device objects
[ https://issues.apache.org/jira/browse/YARN-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883064#comment-16883064 ] Peter Bacsko commented on YARN-9135: OK, so apparently there are no findbugs issues anymore, which is cool. I think this one is good to go. > NM State store ResourceMappings serialization are tested with Strings instead > of real Device objects > > > Key: YARN-9135 > URL: https://issues.apache.org/jira/browse/YARN-9135 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9135.001.patch, YARN-9135.003.patch, > YARN-9135.004.patch, YARN-9135.005.patch, YARN-9135.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883049#comment-16883049 ] Peter Bacsko commented on YARN-9127: XML -> I think that's because of the missing DTD that cannot be found anywhere. It's safe to ignore. Checkstlye -> addressed in patch v5, which is pending. > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch, YARN-9127.004.patch, YARN-9127.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883049#comment-16883049 ] Peter Bacsko edited comment on YARN-9127 at 7/11/19 2:56 PM: - XML -> That's because of the missing DTD that cannot be found anywhere. It's safe to ignore. Checkstlye -> addressed in patch v5, which is pending. was (Author: pbacsko): XML -> I think that's because of the missing DTD that cannot be found anywhere. It's safe to ignore. Checkstlye -> addressed in patch v5, which is pending. > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch, YARN-9127.004.patch, YARN-9127.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883047#comment-16883047 ] Peter Bacsko commented on YARN-9140: [~snemeth] that's patch v3 build result. Not sure if build for v4 is running or not, hopefully it'll be posted soon. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9100) Add tests for GpuResourceAllocator and do minor code cleanup
[ https://issues.apache.org/jira/browse/YARN-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883045#comment-16883045 ] Peter Bacsko commented on YARN-9100: [~snemeth] Please wait for the build result of patch v5. The latest result is about v4. > Add tests for GpuResourceAllocator and do minor code cleanup > > > Key: YARN-9100 > URL: https://issues.apache.org/jira/browse/YARN-9100 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9100-004.patch, YARN-9100-005.patch, > YARN-9100.001.patch, YARN-9100.002.patch, YARN-9100.003.patch > > > Add tests for GpuResourceAllocator and do minor code cleanup > - Improved log and exception messages > - Added some new debug logs > - Some methods are named like *Copy, these are returning copies of internal > data structures. The word "copy" is just a noise in their name, so they have > been renamed. Additionally, the copied data structures modified to be > immutable. > - The waiting loop in method assignGpus were decoupled into a new class, > RetryCommand. > Some more words about the new class RetryCommand: > There are some similar waiting loops in the code in: AMRMClient, > AMRMClientAsync and even in GenericTestUtils (see waitFor method). > RetryCommand could be a future replacement of these duplicated code, as it > gives a solution to this waiting loop problem in a generic way. > The only downside of the usage of RetryCommand in GpuResourceAllocator > (startGpuAssignmentLoop) is the ugly exception handling part, but that's > solely because how Java deals with checked exceptions vs. lambdas. If there's > a cleaner way to solve the exception handling, I'm open for any suggestions. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883041#comment-16883041 ] Szilard Nemeth commented on YARN-9127: -- Hi [~pbacsko]! Could you please take care of the checkstyle and XML issues? Thanks! > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch, YARN-9127.004.patch, YARN-9127.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883039#comment-16883039 ] Szilard Nemeth commented on YARN-9140: -- Hi [~pbacsko]! Could you please check the javac issues? Thanks! > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9100) Add tests for GpuResourceAllocator and do minor code cleanup
[ https://issues.apache.org/jira/browse/YARN-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883037#comment-16883037 ] Szilard Nemeth commented on YARN-9100: -- Hi [~pbacsko]! Could you please fix the remaining checkstyle issues? Thanks! > Add tests for GpuResourceAllocator and do minor code cleanup > > > Key: YARN-9100 > URL: https://issues.apache.org/jira/browse/YARN-9100 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9100-004.patch, YARN-9100-005.patch, > YARN-9100.001.patch, YARN-9100.002.patch, YARN-9100.003.patch > > > Add tests for GpuResourceAllocator and do minor code cleanup > - Improved log and exception messages > - Added some new debug logs > - Some methods are named like *Copy, these are returning copies of internal > data structures. The word "copy" is just a noise in their name, so they have > been renamed. Additionally, the copied data structures modified to be > immutable. > - The waiting loop in method assignGpus were decoupled into a new class, > RetryCommand. > Some more words about the new class RetryCommand: > There are some similar waiting loops in the code in: AMRMClient, > AMRMClientAsync and even in GenericTestUtils (see waitFor method). > RetryCommand could be a future replacement of these duplicated code, as it > gives a solution to this waiting loop problem in a generic way. > The only downside of the usage of RetryCommand in GpuResourceAllocator > (startGpuAssignmentLoop) is the ugly exception handling part, but that's > solely because how Java deals with checked exceptions vs. lambdas. If there's > a cleaner way to solve the exception handling, I'm open for any suggestions. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9135) NM State store ResourceMappings serialization are tested with Strings instead of real Device objects
[
https://issues.apache.org/jira/browse/YARN-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883013#comment-16883013
]
Hadoop QA commented on YARN-9135:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
9s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9135 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974403/YARN-9135.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux aae973e4cd1a 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / acef5e0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24372/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24372/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> NM State store ResourceMappings serialization are tested with
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883001#comment-16883001 ] Szilard Nemeth commented on YARN-9625: -- Hi [~zsiegl]! Thanks for the latest patch, committed to trunk! Pending Jenkins on branch-3.2 / branch-3.1 patches. > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch, YARN-9625.branch-3.2.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9625: - Attachment: YARN-9625.branch-3.2.001.patch > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch, YARN-9625.branch-3.2.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882997#comment-16882997 ] Hudson commented on YARN-9625: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16889 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16889/]) YARN-9625. UI2 - No link to a queue on the Queues page for Fair (snemeth: rev 9cec02318644c8430cbf65bcc3096ffe45992a8e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/components/yarn-queue/fair-queue.hbs > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9127: --- Attachment: YARN-9127.005.patch > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch, YARN-9127.004.patch, YARN-9127.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[
https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882992#comment-16882992
]
Hadoop QA commented on YARN-9127:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 8s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 5 unchanged - 5 fixed = 7 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
40s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 7s{color} |
{color:red} The patch has 4 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
9s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
48s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
|
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882977#comment-16882977 ] Szilard Nemeth commented on YARN-9625: -- Okay, I see your points [~zsiegl]! Committing to trunk soon! > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882958#comment-16882958 ] Zoltan Siegl commented on YARN-9625: Yes, indeed the CS page is different to FS page. The CSS class does not apply to FS page. I don't believe that h3 should be used for this link as FS page uses a slightly different site build, and also h3 is semantically totally incorrect to use here. It is not a heading of any type. > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6715) Fix documentation about NodeHealthScriptRunner
[
https://issues.apache.org/jira/browse/YARN-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-6715:
-
Description:
NodeHealthScriptRunner does *not* report a bad health if the script exits with
an exit code other than 0. Look at the {{FAILED_WITH_EXIT_CODE}} case:
{noformat}
void reportHealthStatus(HealthCheckerExitStatus status) {
long now = System.currentTimeMillis();
switch (status) {
case SUCCESS:
setHealthStatus(true, "", now);
break;
case TIMED_OUT:
setHealthStatus(false, NODE_HEALTH_SCRIPT_TIMED_OUT_MSG);
break;
case FAILED_WITH_EXCEPTION:
setHealthStatus(false, exceptionStackTrace);
break;
case FAILED_WITH_EXIT_CODE:
setHealthStatus(true, "", now);
break;
case FAILED:
setHealthStatus(false, shexec.getOutput());
break;
}
}
{noformat}
Based on the discussion in YARN-5567, this is intentional, but conflicts with
the upstream document, which says:
"If the script *exits with a non-zero exit code*, times out or results in an
exception being thrown, the node is marked as unhealthy"
This statement can be extremely misleading and must be corrected. We might also
add an extra comment to {{reportHealthStatus()}} which explains that
{{FAILED_WITH_EXIT_CODE}} is not buggy.
This case also lacks unit test coverage.
was:
NodeHealthScriptRunner does *not* report a bad health if the script exits with
an exit code other than 0. Look at the {{FAILED_WITH_EXIT_CODE}} case:
{noformat}
void reportHealthStatus(HealthCheckerExitStatus status) {
long now = System.currentTimeMillis();
switch (status) {
case SUCCESS:
setHealthStatus(true, "", now);
break;
case TIMED_OUT:
setHealthStatus(false, NODE_HEALTH_SCRIPT_TIMED_OUT_MSG);
break;
case FAILED_WITH_EXCEPTION:
setHealthStatus(false, exceptionStackTrace);
break;
case FAILED_WITH_EXIT_CODE:
setHealthStatus(true, "", now);
break;
case FAILED:
setHealthStatus(false, shexec.getOutput());
break;
}
}
{noformat}
Based on the discussion in YARN-5567, this is intional, but conflicts with the
upstream document, which says:
"If the script *exits with a non-zero exit code*, times out or results in an
exception being thrown, the node is marked as unhealthy"
This statement can be extremely misleading and must be corrected. We might also
add an extra comment to {{reportHealthStatus()}} which explains that
{{FAILED_WITH_EXIT_CODE}} is not buggy.
This case also lacks unit test coverage.
> Fix documentation about NodeHealthScriptRunner
> ---
>
> Key: YARN-6715
> URL: https://issues.apache.org/jira/browse/YARN-6715
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> NodeHealthScriptRunner does *not* report a bad health if the script exits
> with an exit code other than 0. Look at the {{FAILED_WITH_EXIT_CODE}} case:
> {noformat}
> void reportHealthStatus(HealthCheckerExitStatus status) {
> long now = System.currentTimeMillis();
> switch (status) {
> case SUCCESS:
> setHealthStatus(true, "", now);
> break;
> case TIMED_OUT:
> setHealthStatus(false, NODE_HEALTH_SCRIPT_TIMED_OUT_MSG);
> break;
> case FAILED_WITH_EXCEPTION:
> setHealthStatus(false, exceptionStackTrace);
> break;
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> case FAILED:
> setHealthStatus(false, shexec.getOutput());
> break;
> }
> }
> {noformat}
> Based on the discussion in YARN-5567, this is intentional, but conflicts with
> the upstream document, which says:
> "If the script *exits with a non-zero exit code*, times out or results in an
> exception being thrown, the node is marked as unhealthy"
> This statement can be extremely misleading and must be corrected. We might
> also add an extra comment to {{reportHealthStatus()}} which explains that
> {{FAILED_WITH_EXIT_CODE}} is not buggy.
> This case also lacks unit test coverage.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[
https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882949#comment-16882949
]
Szilard Nemeth commented on YARN-9626:
--
Hi [~zsiegl]!
Changes in the patch looks good, however, Jenkins could not apply patch to
trunk.
I think the reason for this is the following:
{code:java}
??-( szilardnemeth@snemeth-MBP[15:21:25] <0> @trunk )--(
~/development/apache/hadoop-commit )--
└-$ git apply ~/Downloads/YARN-9626.001.patch
error:
hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/components/yarn-queue/fair-queue-conf-table.hbs:
No such file or directory
error: hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-queue.hbs:
No such file or directory
??-( szilardnemeth@snemeth-MBP[15:22:22] >1< @trunk )--(
~/development/apache/hadoop-commit )--
└-$ git apply ~/Downloads/YARN-9626.001.patch -p0
??-( szilardnemeth@snemeth-MBP[15:22:29] <0> @trunk )--(
~/development/apache/hadoop-commit )--
└-$ git st
On branch trunk
Your branch is up to date with 'origin/trunk'.
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)
modified:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/components/yarn-queue/fair-queue-conf-table.hbs
modified:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-queue.hbs
no changes added to commit (use "git add" and/or "git commit -a")
??-( szilardnemeth@snemeth-MBP[15:22:30] <0> @trunk )--(
~/development/apache/hadoop-commit )--
{code}
I can see that you generated the patch with intellij, please export the patch
again by simply issuing:
{code:java}
git diff >
{code}
> UI2 - Fair scheduler queue apps page issues
> ---
>
> Key: YARN-9626
> URL: https://issues.apache.org/jira/browse/YARN-9626
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: Fair_scheduler_apps_page.png, YARN-9626.001.patch,
> YARN-9626_after_patch.png
>
>
> There are a few issues with the apps page for a queue when Fair Scheduler is
> used.
> * Labels like configured capacity, configured max capacity etc. (marked in
> the attached image) are not needed as they are specific to Capacity Scheduler.
> * Steady fair memory, used memory and maximum memory are actual values but
> are shown as percentages.
> * Formatting of Pending, Allocated, Reserved Containers values is not
> correct (shown in the attached screenshot)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882931#comment-16882931 ] Szilard Nemeth commented on YARN-9625: -- Hi [~zsiegl]! Checked your patch. I can see that the page for capacity scheduler is a bit different, it has the link inside a h3 HTML tag and a CSS class for the div is also different (yarn-compose-box yarn-queues-container). Isn't it better to reuse the CSS class and have the h3 as well? See the code of capacity-scheduler page for reference: https://github.com/apache/hadoop/blob/acef5e0cec944d4e1d11541372f85487ec4d1d81/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/components/yarn-queue/capacity-queue.hbs#L22-L36 Thanks! > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9135) NM State store ResourceMappings serialization are tested with Strings instead of real Device objects
[ https://issues.apache.org/jira/browse/YARN-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882936#comment-16882936 ] Peter Bacsko commented on YARN-9135: Reuploaded patch v5 so that we can see the findbugs problems again. > NM State store ResourceMappings serialization are tested with Strings instead > of real Device objects > > > Key: YARN-9135 > URL: https://issues.apache.org/jira/browse/YARN-9135 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9135.001.patch, YARN-9135.003.patch, > YARN-9135.004.patch, YARN-9135.005.patch, YARN-9135.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9135) NM State store ResourceMappings serialization are tested with Strings instead of real Device objects
[ https://issues.apache.org/jira/browse/YARN-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9135: --- Attachment: YARN-9135.006.patch > NM State store ResourceMappings serialization are tested with Strings instead > of real Device objects > > > Key: YARN-9135 > URL: https://issues.apache.org/jira/browse/YARN-9135 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9135.001.patch, YARN-9135.003.patch, > YARN-9135.004.patch, YARN-9135.005.patch, YARN-9135.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9625: --- Attachment: YARN-9625-after-patch.png > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[
https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882922#comment-16882922
]
Hadoop QA commented on YARN-9625:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-9625 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9625 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24370/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 - No link to a queue on the Queues page for Fair Scheduler
> --
>
> Key: YARN-9625
> URL: https://issues.apache.org/jira/browse/YARN-9625
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png,
> YARN-9625-after-patch.png, YARN-9625.001.patch
>
>
> When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab
> on the right with a link to a certain queue which provides running app
> information for the queue. But for 'Fair Scheduler' there is no such link.
> Attached screenshots for both schedulers.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[
https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882923#comment-16882923
]
Hadoop QA commented on YARN-9626:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-9626 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9626 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24371/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 - Fair scheduler queue apps page issues
> ---
>
> Key: YARN-9626
> URL: https://issues.apache.org/jira/browse/YARN-9626
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: Fair_scheduler_apps_page.png, YARN-9626.001.patch,
> YARN-9626_after_patch.png
>
>
> There are a few issues with the apps page for a queue when Fair Scheduler is
> used.
> * Labels like configured capacity, configured max capacity etc. (marked in
> the attached image) are not needed as they are specific to Capacity Scheduler.
> * Steady fair memory, used memory and maximum memory are actual values but
> are shown as percentages.
> * Formatting of Pending, Allocated, Reserved Containers values is not
> correct (shown in the attached screenshot)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[ https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9626: --- Attachment: (was: Screenshot 2019-07-11 at 14.25.30.png) > UI2 - Fair scheduler queue apps page issues > --- > > Key: YARN-9626 > URL: https://issues.apache.org/jira/browse/YARN-9626 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Fair_scheduler_apps_page.png, YARN-9626.001.patch > > > There are a few issues with the apps page for a queue when Fair Scheduler is > used. > * Labels like configured capacity, configured max capacity etc. (marked in > the attached image) are not needed as they are specific to Capacity Scheduler. > * Steady fair memory, used memory and maximum memory are actual values but > are shown as percentages. > * Formatting of Pending, Allocated, Reserved Containers values is not > correct (shown in the attached screenshot) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9625: --- Attachment: (was: fair_scheduler-page_after_patch.png) > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625-after-patch.png, YARN-9625.001.patch > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[ https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9626: --- Attachment: YARN-9626_after_patch.png > UI2 - Fair scheduler queue apps page issues > --- > > Key: YARN-9626 > URL: https://issues.apache.org/jira/browse/YARN-9626 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Fair_scheduler_apps_page.png, YARN-9626.001.patch, > YARN-9626_after_patch.png > > > There are a few issues with the apps page for a queue when Fair Scheduler is > used. > * Labels like configured capacity, configured max capacity etc. (marked in > the attached image) are not needed as they are specific to Capacity Scheduler. > * Steady fair memory, used memory and maximum memory are actual values but > are shown as percentages. > * Formatting of Pending, Allocated, Reserved Containers values is not > correct (shown in the attached screenshot) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[
https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882913#comment-16882913
]
Hadoop QA commented on YARN-9626:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red} YARN-9626 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9626 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24369/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 - Fair scheduler queue apps page issues
> ---
>
> Key: YARN-9626
> URL: https://issues.apache.org/jira/browse/YARN-9626
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: Fair_scheduler_apps_page.png, Screenshot 2019-07-11 at
> 14.25.30.png, YARN-9626.001.patch
>
>
> There are a few issues with the apps page for a queue when Fair Scheduler is
> used.
> * Labels like configured capacity, configured max capacity etc. (marked in
> the attached image) are not needed as they are specific to Capacity Scheduler.
> * Steady fair memory, used memory and maximum memory are actual values but
> are shown as percentages.
> * Formatting of Pending, Allocated, Reserved Containers values is not
> correct (shown in the attached screenshot)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[
https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882912#comment-16882912
]
Hadoop QA commented on YARN-9625:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-9625 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9625 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24368/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 - No link to a queue on the Queues page for Fair Scheduler
> --
>
> Key: YARN-9625
> URL: https://issues.apache.org/jira/browse/YARN-9625
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png,
> YARN-9625.001.patch, fair_scheduler-page_after_patch.png
>
>
> When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab
> on the right with a link to a certain queue which provides running app
> information for the queue. But for 'Fair Scheduler' there is no such link.
> Attached screenshots for both schedulers.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-9670) Missing Fsync for localized resources before updating to finalized in statestore
[ https://issues.apache.org/jira/browse/YARN-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Filipiak updated YARN-9670: --- Comment: was deleted (was: it bites us in YARN 2.6 i scanned through master briefly and couldn't find anything that would fix it.) > Missing Fsync for localized resources before updating to finalized in > statestore > > > Key: YARN-9670 > URL: https://issues.apache.org/jira/browse/YARN-9670 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jan Filipiak >Priority: Major > > A resource that was localized is not properly FSynced before the > state-manager is updated to track this resource as finalized. The Download is > currently considered finished after the target local outputstream is closed. > The data might not have made it to the blockdevice before the statestore is > updated. Containers relying on the resource may see only parts of the > resource after recovery usually leading to them crashing. > > Possible fixes: > Introduce a new step in the state machine that Fsyncs the downloaded path > before calling the statestore. > On recovery we can compare the size (and we probably have to unpack archives > again) > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882911#comment-16882911 ] Zoltan Siegl commented on YARN-9625: Attached screenshot (fair_scheduler-page_after_patch) to show changes. > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625.001.patch, fair_scheduler-page_after_patch.png > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9625) UI2 - No link to a queue on the Queues page for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9625: --- Attachment: fair_scheduler-page_after_patch.png > UI2 - No link to a queue on the Queues page for Fair Scheduler > -- > > Key: YARN-9625 > URL: https://issues.apache.org/jira/browse/YARN-9625 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Capacity_scheduler_page.png, Fair_scheduler_page.png, > YARN-9625.001.patch, fair_scheduler-page_after_patch.png > > > When the scheduler is set as 'Capacity Scheduler' the Queues page has a tab > on the right with a link to a certain queue which provides running app > information for the queue. But for 'Fair Scheduler' there is no such link. > Attached screenshots for both schedulers. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[ https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882909#comment-16882909 ] Zoltan Siegl commented on YARN-9626: Hi [~snemeth]! Thank you for checking this. Screenshot attached. > UI2 - Fair scheduler queue apps page issues > --- > > Key: YARN-9626 > URL: https://issues.apache.org/jira/browse/YARN-9626 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Fair_scheduler_apps_page.png, Screenshot 2019-07-11 at > 14.25.30.png, YARN-9626.001.patch > > > There are a few issues with the apps page for a queue when Fair Scheduler is > used. > * Labels like configured capacity, configured max capacity etc. (marked in > the attached image) are not needed as they are specific to Capacity Scheduler. > * Steady fair memory, used memory and maximum memory are actual values but > are shown as percentages. > * Formatting of Pending, Allocated, Reserved Containers values is not > correct (shown in the attached screenshot) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[ https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9626: --- Attachment: Screenshot 2019-07-11 at 14.25.30.png > UI2 - Fair scheduler queue apps page issues > --- > > Key: YARN-9626 > URL: https://issues.apache.org/jira/browse/YARN-9626 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Fair_scheduler_apps_page.png, Screenshot 2019-07-11 at > 14.25.30.png, YARN-9626.001.patch > > > There are a few issues with the apps page for a queue when Fair Scheduler is > used. > * Labels like configured capacity, configured max capacity etc. (marked in > the attached image) are not needed as they are specific to Capacity Scheduler. > * Steady fair memory, used memory and maximum memory are actual values but > are shown as percentages. > * Formatting of Pending, Allocated, Reserved Containers values is not > correct (shown in the attached screenshot) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[ https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882903#comment-16882903 ] Szilard Nemeth edited comment on YARN-9626 at 7/11/19 12:26 PM: Hi [~zsiegl]! On the screenshot, the section of "Pending, Allocated and Reserved" containers should show 3 numbers, right? Is this fixed with this patch? Could you please update a new screenshot that is generated by the code with your changes? Thanks! was (Author: snemeth): Hi [~zsiegl]! Thanks for the screenshot, it looks good. However, the section of "Pending, Allocated and Reserved" containers should show 3 numbers, right? On the screenshot, I can only see 2 numbers and a comma. This looks weird, so please fix it! Thanks! > UI2 - Fair scheduler queue apps page issues > --- > > Key: YARN-9626 > URL: https://issues.apache.org/jira/browse/YARN-9626 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Fair_scheduler_apps_page.png, YARN-9626.001.patch > > > There are a few issues with the apps page for a queue when Fair Scheduler is > used. > * Labels like configured capacity, configured max capacity etc. (marked in > the attached image) are not needed as they are specific to Capacity Scheduler. > * Steady fair memory, used memory and maximum memory are actual values but > are shown as percentages. > * Formatting of Pending, Allocated, Reserved Containers values is not > correct (shown in the attached screenshot) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9626) UI2 - Fair scheduler queue apps page issues
[ https://issues.apache.org/jira/browse/YARN-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882903#comment-16882903 ] Szilard Nemeth commented on YARN-9626: -- Hi [~zsiegl]! Thanks for the screenshot, it looks good. However, the section of "Pending, Allocated and Reserved" containers should show 3 numbers, right? On the screenshot, I can only see 2 numbers and a comma. This looks weird, so please fix it! Thanks! > UI2 - Fair scheduler queue apps page issues > --- > > Key: YARN-9626 > URL: https://issues.apache.org/jira/browse/YARN-9626 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Zoltan Siegl >Priority: Major > Attachments: Fair_scheduler_apps_page.png, YARN-9626.001.patch > > > There are a few issues with the apps page for a queue when Fair Scheduler is > used. > * Labels like configured capacity, configured max capacity etc. (marked in > the attached image) are not needed as they are specific to Capacity Scheduler. > * Steady fair memory, used memory and maximum memory are actual values but > are shown as percentages. > * Formatting of Pending, Allocated, Reserved Containers values is not > correct (shown in the attached screenshot) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9670) Missing Fsync for localized resources before updating to finalized in statestore
[ https://issues.apache.org/jira/browse/YARN-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882901#comment-16882901 ] Jan Filipiak commented on YARN-9670: it bites us in YARN 2.6 i scanned through master briefly and couldn't find anything that would fix it. > Missing Fsync for localized resources before updating to finalized in > statestore > > > Key: YARN-9670 > URL: https://issues.apache.org/jira/browse/YARN-9670 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jan Filipiak >Priority: Major > > A resource that was localized is not properly FSynced before the > state-manager is updated to track this resource as finalized. The Download is > currently considered finished after the target local outputstream is closed. > The data might not have made it to the blockdevice before the statestore is > updated. Containers relying on the resource may see only parts of the > resource after recovery usually leading to them crashing. > > Possible fixes: > Introduce a new step in the state machine that Fsyncs the downloaded path > before calling the statestore. > On recovery we can compare the size (and we probably have to unpack archives > again) > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9670) Missing Fsync for localized resources before updating to finalized in statestore
[ https://issues.apache.org/jira/browse/YARN-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882902#comment-16882902 ] Jan Filipiak commented on YARN-9670: it bites us in YARN 2.6 i scanned through master briefly and couldn't find anything that would fix it. > Missing Fsync for localized resources before updating to finalized in > statestore > > > Key: YARN-9670 > URL: https://issues.apache.org/jira/browse/YARN-9670 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jan Filipiak >Priority: Major > > A resource that was localized is not properly FSynced before the > state-manager is updated to track this resource as finalized. The Download is > currently considered finished after the target local outputstream is closed. > The data might not have made it to the blockdevice before the statestore is > updated. Containers relying on the resource may see only parts of the > resource after recovery usually leading to them crashing. > > Possible fixes: > Introduce a new step in the state machine that Fsyncs the downloaded path > before calling the statestore. > On recovery we can compare the size (and we probably have to unpack archives > again) > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9670) Missing Fsync for localized resources before updating to finalized in statestore
[ https://issues.apache.org/jira/browse/YARN-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Filipiak updated YARN-9670: --- Summary: Missing Fsync for localized resources before updating to finalized in statestore (was: Missing Fsync for localized resostatestoreurces before updating to finalized in ) > Missing Fsync for localized resources before updating to finalized in > statestore > > > Key: YARN-9670 > URL: https://issues.apache.org/jira/browse/YARN-9670 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jan Filipiak >Priority: Major > > A resource that was localized is not properly FSynced before the > state-manager is updated to track this resource as finalized. The Download is > currently considered finished after the target local outputstream is closed. > The data might not have made it to the blockdevice before the statestore is > updated. Containers relying on the resource may see only parts of the > resource after recovery usually leading to them crashing. > > Possible fixes: > Introduce a new step in the state machine that Fsyncs the downloaded path > before calling the statestore. > On recovery we can compare the size (and we probably have to unpack archives > again) > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9127: --- Attachment: YARN-9127.004.patch > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch, YARN-9127.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9670) Missing Fsync for localized resostatestoreurces before updating to finalized in
Jan Filipiak created YARN-9670: -- Summary: Missing Fsync for localized resostatestoreurces before updating to finalized in Key: YARN-9670 URL: https://issues.apache.org/jira/browse/YARN-9670 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.6.0 Reporter: Jan Filipiak A resource that was localized is not properly FSynced before the state-manager is updated to track this resource as finalized. The Download is currently considered finished after the target local outputstream is closed. The data might not have made it to the blockdevice before the statestore is updated. Containers relying on the resource may see only parts of the resource after recovery usually leading to them crashing. Possible fixes: Introduce a new step in the state machine that Fsyncs the downloaded path before calling the statestore. On recovery we can compare the size (and we probably have to unpack archives again) -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882900#comment-16882900 ] Peter Bacsko commented on YARN-9127: Patch v4: did a small cleanup at various places. > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch, YARN-9127.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9127: --- Attachment: (was: YARN-9124.004.patch) > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9127) Create more tests to verify GpuDeviceInformationParser
[ https://issues.apache.org/jira/browse/YARN-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9127: --- Attachment: YARN-9124.004.patch > Create more tests to verify GpuDeviceInformationParser > -- > > Key: YARN-9127 > URL: https://issues.apache.org/jira/browse/YARN-9127 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9127.001.patch, YARN-9127.002.patch, > YARN-9127.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9140: --- Attachment: YARN-9140.004.patch > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9100) Add tests for GpuResourceAllocator and do minor code cleanup
[ https://issues.apache.org/jira/browse/YARN-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9100: --- Attachment: YARN-9100-005.patch > Add tests for GpuResourceAllocator and do minor code cleanup > > > Key: YARN-9100 > URL: https://issues.apache.org/jira/browse/YARN-9100 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9100-004.patch, YARN-9100-005.patch, > YARN-9100.001.patch, YARN-9100.002.patch, YARN-9100.003.patch > > > Add tests for GpuResourceAllocator and do minor code cleanup > - Improved log and exception messages > - Added some new debug logs > - Some methods are named like *Copy, these are returning copies of internal > data structures. The word "copy" is just a noise in their name, so they have > been renamed. Additionally, the copied data structures modified to be > immutable. > - The waiting loop in method assignGpus were decoupled into a new class, > RetryCommand. > Some more words about the new class RetryCommand: > There are some similar waiting loops in the code in: AMRMClient, > AMRMClientAsync and even in GenericTestUtils (see waitFor method). > RetryCommand could be a future replacement of these duplicated code, as it > gives a solution to this waiting loop problem in a generic way. > The only downside of the usage of RetryCommand in GpuResourceAllocator > (startGpuAssignmentLoop) is the ugly exception handling part, but that's > solely because how Java deals with checked exceptions vs. lambdas. If there's > a cleaner way to solve the exception handling, I'm open for any suggestions. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
