[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642167#comment-16642167 ] Hudson commented on YARN-8659: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15139 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15139/]) YARN-8659. RMWebServices returns only RUNNING apps when filtered with (haibochen: rev 7c13872cbbb6f1b0b1c2dde894885b41186b3797) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0 > > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch, YARN-8659.004.patch, YARN-8659.005.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642118#comment-16642118 ] Haibo Chen commented on YARN-8659: -- Thanks [~Prabhu Joseph] for the initial report and [~snemeth] for the fix! > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0 > > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch, YARN-8659.004.patch, YARN-8659.005.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642114#comment-16642114 ] Haibo Chen commented on YARN-8659: -- +1 on the latest patch. Will commit it to trunk shortly. > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch, YARN-8659.004.patch, YARN-8659.005.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[
https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640822#comment-16640822
]
Hadoop QA commented on YARN-8659:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 43 unchanged - 8 fixed = 43 total (was 51) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 6s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942680/YARN-8659.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7586fce24f0f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cdf5d58 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22083/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22083/testReport/ |
| Max. process+thread count | 934 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[
https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640632#comment-16640632
]
Hadoop QA commented on YARN-8659:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-8659 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942662/YARN-8659.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22080/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RMWebServices returns only RUNNING apps when filtered with queue
>
>
> Key: YARN-8659
> URL: https://issues.apache.org/jira/browse/YARN-8659
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot
> 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch,
> YARN-8659.003.patch, YARN-8659.004.patch
>
>
> RMWebServices returns only RUNNING apps when filtered with queue and returns
> empty apps
> when filtered with both FINISHED states and queue.
> http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default
> http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640584#comment-16640584 ] Szilard Nemeth commented on YARN-8659: -- patch004 fixes a unit test failre, that was introduced with patch003 > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch, YARN-8659.004.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[
https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640572#comment-16640572
]
Hadoop QA commented on YARN-8659:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 43 unchanged - 8 fixed = 43 total (was 51) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 1s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942653/YARN-8659.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2f9e08d9db34 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cdf5d58 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22078/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22078/testReport/ |
| Max. process+thread count | 896 (vs. ulimit of 1) |
|
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640520#comment-16640520 ] Szilard Nemeth commented on YARN-8659: -- Thanks [~haibochen] for your comments. Both issues are fixed. > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640479#comment-16640479 ] Haibo Chen commented on YARN-8659: -- Thanks [~snemeth] for the update! The new patch looks good to me overall. Two minor comments 1) getAppsByApplicationIds() returns a map of appId to its json object. I think it suffices to just return the set of applciationIds collected, given we are not checking any other app information 2) 'rm.waitForState(app.getApplicationId(), RMAppState.FINISHED);' can be moved to finishApp() method too. The checkstyle issue can also be addressed. > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[
https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640409#comment-16640409
]
Hadoop QA commented on YARN-8659:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 43 unchanged - 8 fixed = 44 total (was 51) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942602/YARN-8659.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 42994b88a9c9 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c968365 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22073/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22073/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[
https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640192#comment-16640192
]
Szilard Nemeth commented on YARN-8659:
--
Thanks [~haibochen] for confirming that this is a bug.
1. This was really missing, added a new testcase for that.
2. This was already covered with testcase {{testAppsQueryQueue2}}.
So to summarize, I have 3 testcases now:
1. {{testAppsQueryQueueAndStateFinished}}: Two applications, both finished,
query with queue specified and state=finished --> the result should contain
both applications
2. {{testAppsQueryQueueAndStateOneFinishedApp}}: Two applications, one running
and the other finished, query with state=finished and a queue --> the result
should contain the finished app
3. {{testAppsQueryQueueOneFinishedApp}}: Two applications, one running and the
other finished, query with queue specified but no state --> the result should
include both applications
> RMWebServices returns only RUNNING apps when filtered with queue
>
>
> Key: YARN-8659
> URL: https://issues.apache.org/jira/browse/YARN-8659
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot
> 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch
>
>
> RMWebServices returns only RUNNING apps when filtered with queue and returns
> empty apps
> when filtered with both FINISHED states and queue.
> http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default
> http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638932#comment-16638932 ] Haibo Chen commented on YARN-8659: -- Thanks [~snemeth] for the analysis. I agree with your observation. When a queue is specified, the matching applications are retrieved currently from the scheduler, which knows only about the ones that are still actively involved in scheduling decision, a subset of all the applications that RM knows about. An application is removed from the scheduler's knowledge as soon as its scheduling part is done, but RM can still retain the application in its memory until it is evicted based on the max # of applications RM is allowed to remember. IMO this is more of a bug than a API-compatibility-breaking change. To reproduce the bug and verify the behavioral change, I think we need two uni test methods, 1) Two applications, one running and the other finished, query with state=finished and a queue, the result should contain the finished app (the result is empty now) 2) Two applications, one running and the other finished, query with queue specified but no state, the result should include both applications (the result is empty now) > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[
https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604911#comment-16604911
]
Hadoop QA commented on YARN-8659:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 44 unchanged - 4 fixed = 52 total (was 48) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 50s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12938519/YARN-8659.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ffa8d4321477 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9af96d4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/21771/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[
https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604755#comment-16604755
]
Szilard Nemeth commented on YARN-8659:
--
Hi [~Prabhu Joseph]!
I found out the root cause of this bug and I was able to reproduce the bug with
2 testcases.
When {{ClientRMService#getApplications}} is invoked, it first checks whether
the user filters for queues. If yes, it iterates over the specified queues and
retrieves the apps bound to the queue from the scheduler. Then, as a last step,
a tricky iterator is set up, that basically can iterate over the collected
application attempt IDs (since we can have multiple queues and each queue can
have many apps associated to it, it's a list of lists).
See the iterator here:
https://github.com/apache/hadoop/blob/9af96d4ed4b6f80d3ca53a2b003d2ef768650dd4/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java#L834-L859
What essentially is broken is the code that gets the application attempt IDs
from the scheduler:
https://github.com/apache/hadoop/blob/9af96d4ed4b6f80d3ca53a2b003d2ef768650dd4/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java#L829
The scheduler only returns the scheduled applications, but not the finished
ones. This essentially means whatever is specified for the application state
parameter, the code would only give applications back that are currently
executing.
Let's go back what you described above:
1. Just the RUNNING apps are returned if any queue is specified because the
call to {{scheduler.getAppsInQueue\(queue\);}} only returns apps that are
executing.
2. No applications are returned if the queue parameter is specified and the
state parameter is set to FINISHED.
As described above, this is faulty even if you don't specify a state parameter
at all, as the call to {{scheduler.getAppsInQueue\(queue\);}} only returns apps
that are executing, but not the other ones.
So basically, the solution for this is removing the tricky iterator and simply
iterate over the apps retrieved from RMContext.
This should really work as the current code is also getting the applications
from that collection:
https://github.com/apache/hadoop/blob/9af96d4ed4b6f80d3ca53a2b003d2ef768650dd4/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java#L850
We should keep in mind API compatibility, though.
With the current implementation, apps are only returned for a queue if they are
executing.
With the code changes in my patch, if the user specifies the queue filter, the
endpoint returns apps regardless of their states.
If we think about the apps endpoint as a set of filter parameters applied on
applications, it seems to be more logical to return apps bound to a queue,
regardless of what states they have, if the only filter is the queue filter.
If the user wants to have the apps that are executing and bound to a queue, one
should specify both the queue and the state parameters.
> RMWebServices returns only RUNNING apps when filtered with queue
>
>
> Key: YARN-8659
> URL: https://issues.apache.org/jira/browse/YARN-8659
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot
> 2018-08-13 at 8.01.52 PM.png
>
>
> RMWebServices returns only RUNNING apps when filtered with queue and returns
> empty apps
> when filtered with both FINISHED states and queue.
> http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default
> http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579919#comment-16579919 ] Prabhu Joseph commented on YARN-8659: - Yes sure [~snemeth] > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
