[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390792#comment-16390792 ] Yufei Gu commented on YARN-5028: It is a nice-to-have one. Trunk is good enough. Thanks [~leftnoteasy]. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390707#comment-16390707 ] Wangda Tan commented on YARN-5028: -- [~yufeigu], since this isn't committed to branch-3.1, just set fix version. Do we need this for 3.1.0? How bad the issue is? > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385855#comment-16385855 ] Gergo Repas commented on YARN-5028: --- Thanks [~yufeigu], [~rohithsharma]! > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382312#comment-16382312 ] Hudson commented on YARN-5028: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13750 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13750/]) YARN-5028. RMStateStore should trim down app state for completed (yufei: rev 6ca4e3017b994a7444b4b8bf936c1d5d19603ccd) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382279#comment-16382279 ] Yufei Gu commented on YARN-5028: Committed the addendum to trunk. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382246#comment-16382246 ] Yufei Gu commented on YARN-5028: The addendum patch looks good to me. +1. This original one only committed to trunk. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382127#comment-16382127 ] Rohith Sharma K S commented on YARN-5028: - I have verified basic RM restart scenarios and looks fine if we include addendum patch. [~sunilg]/[~yufeigu] Could you guys please help to commit this addendum patch? I am not sure which are all the branches this is committed earlier! > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382113#comment-16382113 ] Gergo Repas commented on YARN-5028: --- [~rohithsharma] Apologies for causing the break, and thanks for the addendum patch! The addendum patch looks good to me. (I'm also fine with having my change reverted, and adding it back later after I tested it with ATSv2, even if it's for a later release). > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381950#comment-16381950 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 19s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 39s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}109m 22s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12912576/YARN-5028.007-addendum.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4cac7f56f129 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 55d04a6 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/19858/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19858/testReport/ | | Max. process+thread count | 885 (vs. ulimit of 1) | |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381912#comment-16381912 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 24s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 16s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 47s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}122m 22s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12912576/YARN-5028.007-addendum.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b438bea5a14b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 55d04a6 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/19857/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19857/testReport/ | | Max. process+thread count | 799 (vs. ulimit of 1) | |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381801#comment-16381801 ] Rohith Sharma K S commented on YARN-5028: - Updated patch adding app name and app tags included. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007-addendum.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381784#comment-16381784 ] Rohith Sharma K S commented on YARN-5028: - I found that couple of more issues where app report also throw NPE for following URL i.e *http://localhost:8088/ws/v1/cluster/apps*. It seems we should add application name and application tags as well. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381679#comment-16381679 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 6s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m 35s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}113m 35s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12912541/YARN-5028.007-addendum.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux db3cf93a91a6 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 10663b7 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19852/testReport/ | | Max. process+thread count | 863 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19852/console | | Powered by | Apache Yetus
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381591#comment-16381591 ] Sunil G commented on YARN-5028: --- Addendum patch looks fine. +1 pending jenkins. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381582#comment-16381582 ] Rohith Sharma K S commented on YARN-5028: - As it is blocking ATS enabled recovery flows, taking liberty to attach quick addendum patch for fixing this issue > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, > YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381556#comment-16381556 ] Rohith Sharma K S commented on YARN-5028: - [~grepas] [~yufeigu] this seems to be breaking when ATS is enabled. When applications are recovered for completed applications, if ats is enabled it try to publish which is leading NPE. This NPE causes RM to shutdown which is fatal issue. {noformat} 2018-02-28 03:21:54,322 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(808)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher.appCreated(TimelineServiceV1Publisher.java:91) at org.apache.hadoop.yarn.server.resourcemanager.metrics.CombinedSystemMetricsPublisher.appCreated(CombinedSystemMetricsPublisher.java:44) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.sendATSCreateEvent(RMAppImpl.java:1984) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recover(RMAppImpl.java:946) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:1076) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:1069) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:893) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:361) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:573) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1426) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:803) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1179) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1219) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1215) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320) at org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:893) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:473) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) {noformat} This issue has committed into 3.1 which is blocker in this release. Either this issue need to be reverted or provide a addendum patch for 3.1 release. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372608#comment-16372608 ] Gergo Repas commented on YARN-5028: --- [~yufeigu] Thanks for the commit and reviews. Thanks, but I don't need this change in branch 2. Thanks [~snemeth] and [~rohithsharma] for the reviews. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371928#comment-16371928 ] Hudson commented on YARN-5028: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13697 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13697/]) YARN-5028. RMStateStore should trim down app state for completed (yufei: rev 92cbbfe79ec009a19a71a7f44329a4b2f9fa9be6) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371908#comment-16371908 ] Yufei Gu commented on YARN-5028: Committed to trunk. Thanks for the patch, [~grepas]. Thanks for the review, [~snemeth] and [~rohithsharma]. Do you need a branch-2 version? [~grepas] > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370958#comment-16370958 ] Rohith Sharma K S commented on YARN-5028: - latest patch reasonable to me > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370356#comment-16370356 ] Yufei Gu commented on YARN-5028: +1 for the last patch. Will commit it later. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370203#comment-16370203 ] genericqa commented on YARN-5028: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 50s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 14s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 59s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}110m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12911224/YARN-5028.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c1f290fc82c3 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8896d20 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19746/testReport/ | | Max. process+thread count | 841 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19746/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > RMStateStore should trim down app state for completed
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370093#comment-16370093 ] Gergo Repas commented on YARN-5028: --- 007.patch: I addressed the checkstyle issue, locally ran all unit tests in the RM (Tests run: 2209, Failures: 0, Errors: 0, Skipped: 7). I also tested on a real cluster, and recovery worked with new trimmed-down application state. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369427#comment-16369427 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 29s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 103 unchanged - 0 fixed = 104 total (was 103) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 36s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 36s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}138m 14s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12911135/YARN-5028.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e0c42274ec95 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / b9a429b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19744/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/19744/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369202#comment-16369202 ] Gergo Repas commented on YARN-5028: --- I'm looking into the test failures. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369174#comment-16369174 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 37s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 24s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 76 unchanged - 0 fixed = 77 total (was 76) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 37s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 14s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}110m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore | | | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore | | | hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12911105/YARN-5028.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e0a4253f63d7 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 83e2bb9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19742/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369095#comment-16369095 ] Gergo Repas commented on YARN-5028: --- [~sunilg] - I think we can keep the ACL-s in the same field as currently it's in. I'm clearing the rest of the fields from AMContainerSpec, that's the main change. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369091#comment-16369091 ] Sunil G commented on YARN-5028: --- Thanks [~grepas]. Thats a really good change. Does this mean we have to store ACLs from AMContainerSpec to another field in ApplicationData.? > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369088#comment-16369088 ] Gergo Repas commented on YARN-5028: --- [~rohithsharma] [~yufeigu] - A pi job produced 1891 byte long znode with trunk code, 1866 byte long znode with trunk+YARN-5028.004.patch. Because of the above results I created YARN-5028.005.patch which keeps only the ACL information from the AMContainerSpec, this patch produced a 200 byte long znode for the same pi job. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, > YARN-5028.005.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367864#comment-16367864 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 37s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 22s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 11s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}109m 4s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910901/YARN-5028.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 76f1396301fe 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4c2119f | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19729/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367565#comment-16367565 ] Gergo Repas commented on YARN-5028: --- [~rohithsharma] - Sure, I'm going to put together statistics on znode data sizes with/without the patch. In the earlier patch versions AMContainerSpec was also removed from the submission context, but it was put back in because the ACLs for the applications are coming from the AMContainerSpec - maybe a trimmed down AMContainerSpec containing only ACL data could help - I'll also include such a version in the comparison. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367486#comment-16367486 ] Rohith Sharma K S commented on YARN-5028: - Apologies for pitching in late! thanks [~grepas] for working on this JIRA. IIUC, the primary reason for this JIRA is reduce size of finished application data being stored into RMStateStore. YARN-65 already reduces memory footprint for complected applications after recovery. Most of the data resides on ApplicationSubmissionContext#AMContainerSpec. But current patch is NOT removing AMContainerSpec from submission context. This brings me a doubt that how much memory is being reduced storing from RMStateStore? Do you have any statical comparison before and after the patch? > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367317#comment-16367317 ] Gergo Repas commented on YARN-5028: --- I fixed the bug in 003.patch. The latest run was marked as failed too (There was a timeout or other error in the fork), but there were 2208 tests run, 0 failure, 0 error, 7 skipped - [~yufeigu] I think it's not a real unit test failure, could you please rerun this build? > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366897#comment-16366897 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 15s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 25s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 16s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 11s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}123m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910901/YARN-5028.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c9da2dc169af 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a1e05e0 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19723/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/19723/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366187#comment-16366187 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 40s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 25s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 7s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 27s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}105m 46s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage | | | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore | | | hadoop.yarn.server.resourcemanager.TestApplicationCleanup | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | | | hadoop.yarn.server.resourcemanager.TestRMRestart | | | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA | | | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart | | | hadoop.yarn.server.resourcemanager.TestRMHAForAsyncScheduler | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910758/YARN-5028.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f91293d1c34c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365852#comment-16365852 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 38s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 34m 5s{color} | {color:red} root in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 37s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 33s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 1m 15s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 13s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 13s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 13s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 12s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 101 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 4s{color} | {color:red} The patch 1152 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 0m 56s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 15s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 14s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:blue}0{color} | {color:blue} asflicense {color} | {color:blue} 0m 14s{color} | {color:blue} ASF License check generated no output? {color} | | {color:black}{color} | {color:black} {color} | {color:black} 40m 45s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910758/YARN-5028.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 36c09b87c4f5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / b27ab7d | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | mvninstall |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365788#comment-16365788 ] Gergo Repas commented on YARN-5028: --- [~yufeigu] Thanks for the suggestions, I addressed all of them in v003 (the above two points in the test, removing the null-check around setting AMContainerSpec, and copying the whole AMContainerSpec over into the new app.submission context). > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch, YARN-5028.003.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365047#comment-16365047 ] Yufei Gu commented on YARN-5028: In case of {{createAndPopulateNewRMApp}}, it uses the ApplicationACLs in submission context, if you replace the original one with a new object. There is a risk to load wrong ACL. Nits in the unit test: # The first set of assert seems unnecessary to me. {code} assertEquals(appId, ctx.getApplicationId()); assertEquals("a_queue", ctx.getQueue()); assertEquals(Collections.singletonList("a_command"), ctx.getAMContainerSpec().getCommands()); assertEquals(17L, ctx.getResource().getMemorySize()); assertEquals(schedulingPropertiesMap, ctx.getApplicationSchedulingPropertiesMap()); {code} If you think it is necessary, please add some comments. # For the second set, I think one assert should be enough. Something like this: {code} assertEquals("No prune if application state isn't FINISHED", schedulingPropertiesMap, ctx.getApplicationSchedulingPropertiesMap()); {code} Otherwise, looks good to me. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364266#comment-16364266 ] Gergo Repas commented on YARN-5028: --- [~yufeigu] - Right, the null check can be abandoned (I'll upload a new patch when the rest of the questions are cleared). Regarding your second question: I'm using a new instance to trim down the state as much as possible (also, I had to provide a non-null reference, otherwise an exception is thrown at the line you quoted from createAndPopulateNewRMApp()). > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362817#comment-16362817 ] Yufei Gu commented on YARN-5028: Thanks [~grepas]. Yes. We do need these two fields. {code} this.applicationACLsManager.addApplication(applicationId, submissionContext.getAMContainerSpec().getApplicationACLs()); {code} The code above in method {{createAndPopulateNewRMApp}} shows that getAMContainerSpec() shouldn't be null, which leads to one question in the patch: {code} if (srcCtx.getAMContainerSpec() != null) { context.setAMContainerSpec(new ContainerLaunchContextPBImpl()); } {code} If srcCtx.getAMContainerSpec() is null, then context.getAMContainerSpec() is null, which causes NPE. Another question is why use a new AMContainerSpec instance for context instead of using the one of srcCtx? > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362390#comment-16362390 ] Gergo Repas commented on YARN-5028: --- [~yufeigu] The two fields that are needed for the unit tests to pass are: # AMContainerResourceRequests ** needed for multiple TestWorkPreservingRMRestart tests ** if this field is not present the following exception is being thrown: [https://github.com/apache/hadoop/blob/0c5d7d71a80bccd4ad7eab269d0727b999606a7e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java#L494] # ApplicationType ** needed for TestRMRestart ** the exception when this field is not present: {code}java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.toLowerCase(StringUtils.java:1127) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:869) at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:1017) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {code} > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361601#comment-16361601 ] Yufei Gu commented on YARN-5028: Thanks [~grepas]. I'm OK with the first four fields. Can you tell me a little more about the last two fields? Which unit tests failed? We probably need to change unit tests if the code only breaks unit tests. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360528#comment-16360528 ] Gergo Repas commented on YARN-5028: --- [~yufeigu] Yes, sure - I started out with an empty submission context, but the recovery failed at certain points (when running on a real cluster). From the stack traces it was pretty clear which field was missing for the recovery, and added those to the context until recovery succeeded without any issue - this covers the first four fields of the quoted section. The last two fields were needed for all unit tests to pass. Please let me know if you need the exact stack traces where these fields are needed, it takes some time to produce those so I'll wait for your reply. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358801#comment-16358801 ] Yufei Gu commented on YARN-5028: Hi [~grepas], thanks for working on this. Could you elaborate on the decision on why these fields are kept and not others? {code} // most fields in the ApplicationSubmissionContex are not needed, // but the following few need to be present for recovery to succeed context.setApplicationId(srcCtx.getApplicationId()); context.setResource(srcCtx.getResource()); context.setQueue(srcCtx.getQueue()); context.setAMContainerResourceRequests( srcCtx.getAMContainerResourceRequests()); context.setApplicationType(srcCtx.getApplicationType()); if (srcCtx.getAMContainerSpec() != null) { context.setAMContainerSpec(new ContainerLaunchContextPBImpl()); } {code} > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355237#comment-16355237 ] Szilard Nemeth commented on YARN-5028: -- Hey [~grepas]! I just checked your changes, +1 (non-binding). > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354169#comment-16354169 ] Gergo Repas commented on YARN-5028: --- The TestRMWebServiceAppsNodelabel unit test failure is unrelated (fails on trunk too). > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch, > YARN-5028.002.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352287#comment-16352287 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 4s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 54s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 48s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}122m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909200/YARN-5028.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux bfabe6b18699 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4e9a59c | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/19600/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19600/testReport/ | | Max. process+thread count | 880 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U:
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350701#comment-16350701 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 57s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 45s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 27s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 11s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}125m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel | | | hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementProcessor | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908982/YARN-5028.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 119bf6b51613 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4aef8bd | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19580/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350349#comment-16350349 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 5m 11s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908982/YARN-5028.001.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19577/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Gergo Repas >Priority: Major > Attachments: YARN-5028.000.patch, YARN-5028.001.patch > > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349286#comment-16349286 ] genericqa commented on YARN-5028: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 39s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 5s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 48s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}109m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHAForAsyncScheduler | | | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart | | | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel | | | hadoop.yarn.server.resourcemanager.TestRMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-5028 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908846/YARN-5028.000.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e9d06c47935d 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 6ca7204 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/19568/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results |
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[ https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343593#comment-16343593 ] Gergo Repas commented on YARN-5028: --- Hi [~templedf] - can I please pick this ticket up? > RMStateStore should trim down app state for completed applications > -- > > Key: YARN-5028 > URL: https://issues.apache.org/jira/browse/YARN-5028 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton >Priority: Major > > RMStateStore stores enough information to recover applications in case of a > restart. The store also retains this information for completed applications > to serve their status to REST, WebUI, Java and CLI clients. We don't need all > the information we store today to serve application status; for instance, we > don't need the {{ApplicationSubmissionContext}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org