[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130479#comment-14130479 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668114/YARN-2033.13.patch against trunk revision bf64fce. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4893//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4893//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, > YARN-2033.12.patch, YARN-2033.13.patch, YARN-2033.13.patch, > YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, > YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, YARN-2033.9.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130292#comment-14130292 ] Junping Du commented on YARN-2033: -- Seems not work. [~zjshen], can you provide a new one here? Thanks! > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, > YARN-2033.12.patch, YARN-2033.13.patch, YARN-2033.2.patch, YARN-2033.3.patch, > YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130058#comment-14130058 ] Hadoop QA commented on YARN-2033: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668076/YARN-2033.13.patch against trunk revision 4be9517. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4892//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, > YARN-2033.12.patch, YARN-2033.13.patch, YARN-2033.2.patch, YARN-2033.3.patch, > YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129354#comment-14129354 ] Junping Du commented on YARN-2033: -- Sure. Will commit it soon. Thanks [~zjshen] for the patch and [~vinodkv] for review! > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, > YARN-2033.12.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, > YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129052#comment-14129052 ] Vinod Kumar Vavilapalli commented on YARN-2033: --- +1, looks good to me. [~djp], can you please do the honours given you did early reviews? > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, > YARN-2033.12.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, > YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129042#comment-14129042 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667818/YARN-2033.12.patch against trunk revision 47bdfa0. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4874//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4874//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, > YARN-2033.12.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, > YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129010#comment-14129010 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667808/YARN-2033.11.patch against trunk revision 47bdfa0. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4873//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4873//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, > YARN-2033.12.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, > YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128877#comment-14128877 ] Vinod Kumar Vavilapalli commented on YARN-2033: --- Looks mostly good. Rename yarn.resourcemanager.metrics-publisher.enabled -> also to say system-metrics-publisher too? Similarly rename yarn.resourcemanager.metrics-publisher.dispatcher.pool-size? > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.2.patch, YARN-2033.3.patch, > YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127384#comment-14127384 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667445/YARN-2033.10.patch against trunk revision 90c8ece. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4858//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4858//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.2.patch, YARN-2033.3.patch, > YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126481#comment-14126481 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667301/YARN-2033.9.patch against trunk revision 7498dd7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4852//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4852//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, > YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126340#comment-14126340 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667269/YARN-2033.8.patch against trunk revision d989ac0. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4851//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4851//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126236#comment-14126236 ] Zhijie Shen commented on YARN-2033: --- [~vinodkv], thanks for your comments. I've updated the patch accordingly. bq. RMApplicationHistoryWriter is not really needed anymore. We did document it to be unstable/alpha too. We can remove it directly instead of deprecating it. It's a burden to support two interface hierarchies. I'm okay doing it separately though. Seems to make sense. Previously I created a ticket for deprecating the old history store stack. Let me update that jira. bq. YarnClientImpl: Calls using AHSClient shouldn't rely on timeline-publisher yet, we should continue to use APPLICATION_HISTORY_ENABLED for that till we get rid of AHSClient altogether. We should file a ticket for this too. In the newer patch, I revert the change in YarnClientImpl, making it use APPLICATION_HISTORY_ENABLED. And ApplicationHistoryServer checks APPLICATION_HISTORY_STORE for backward compatibility. This can be simplified once the old history store stack is removed. Also I simplify the configuration check in SystemMetricsPublisher. I'll create a jira for getting rid of AHSClient. bq. You removed the unstable annotations from ApplicationContext APIs. We should retain them, this stuff isn't stable yet. ApplicationContext is for internal usage only, not user-faced interface. So I think it should be removed not to confuse people. bq. Rename YarnMetricsPublisher -> {Platform|System} MetricsPublisher to avoid confusing it with host/daemon metrics that exist outside today? Renamed all yarnmetrics -> systemmetrics. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126073#comment-14126073 ] Vinod Kumar Vavilapalli commented on YARN-2033: --- Mostly looks fine, this is a rapidly changing part of the code-base! I get a feeling we need some umbrella cleanup effort to make consistent usage w.r.t history-service/timeline-service. Anyways, some comments - RMApplicationHistoryWriter is not really needed anymore. We did document it to be unstable/alpha too. We can remove it directly instead of deprecating it. It's a burden to support two interface hierarchies. I'm okay doing it separately though. - YarnClientImpl: Calls using AHSClient shouldn't rely on timeline-publisher yet, we should continue to use APPLICATION_HISTORY_ENABLED for that till we get rid of AHSClient altogether. We should file a ticket for this too. - You removed the unstable annotations from ApplicationContext APIs. We should retain them, this stuff isn't stable yet. - Rename YarnMetricsPublisher -> {Platform|System}MetricsPublisher to avoid confusing it with host/daemon metrics that exist outside today? > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118563#comment-14118563 ] Vinod Kumar Vavilapalli commented on YARN-2033: --- Having a quick look at this.. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117458#comment-14117458 ] Junping Du commented on YARN-2033: -- +1. Latest patch LGTM. Will commit it tomorrow if no new comments from others. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116820#comment-14116820 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665205/YARN-2033.7.patch against trunk revision 258c7d0. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4793//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4793//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114913#comment-14114913 ] Zhijie Shen commented on YARN-2033: --- The test failure should not be related. It seems that the configuration resource was not read correctly on jenkins: {code} java.lang.RuntimeException: java.util.zip.ZipException: oversubscribed dynamic bit lengths tree at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:147) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105) at java.io.FilterInputStream.read(FilterInputStream.java:66) at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source) at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source) at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:153) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2334) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2322) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2393) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2346) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2263) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:605) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247) at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:296) at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.setup(TestRMRestart.java:119) {code} > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114876#comment-14114876 ] Hadoop QA commented on YARN-2033: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665205/YARN-2033.7.patch against trunk revision d8774cc. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4765//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4765//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111015#comment-14111015 ] Junping Du commented on YARN-2033: -- [~zjshen], the patch looks good to me in over all. I don't have much more comments, except below. Would you fix it? bq. First of all, two queries are not duplicate: one to read the application entity, and the other to read the app attempt entity, and we previously distinguish ApplicationNotFoundException and ApplicationAttemptNotFoundException. It is always possible that App1 exists in the store with the only attempt AppAttempt1 while the user looks up for AppAttempt2. In this case, we know App1 is there, but AppAttempt2 isn't, so we will throw ApplicationAttemptNotFoundException. If we really want to differ the two exceptions, we still can check ApplicationAttempt first and check Application later (to see if throw ApplicationNotFoundException instead) if ApplicationAttemptNotFoundException get thrown there. This is more efficient as we only need to visit DB one time in most cases. Isn't it? > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100873#comment-14100873 ] Zhijie Shen commented on YARN-2033: --- The test failures seem to be related to RM HA. bq. IMO, this is not necessary as application should exist in most cases and we don't need to duplicated visit LevelDB twice. If application doesn't exist, we can throw ApplicationNotFoundException in retrieving app attempt info. Isn't it? First of all, two queries are not duplicate: one to read the application entity, and the other to read the app attempt entity, and we previously distinguish ApplicationNotFoundException and ApplicationAttemptNotFoundException. It is always possible that App1 exists in the store with the only attempt AppAttempt1 while the user looks up for AppAttempt2. In this case, we know App1 is there, but AppAttempt2 isn't, so we will throw ApplicationAttemptNotFoundException. Moreover, when we go on with generic history ACLs, we will anyway visit the app entity once to pull the user info for access check. bq. The point here is we should check on the combination of related configurations and make all wrong combinations get warned. Any concern on doing this? Right, so in the new patch, I've enhanced the configuration check logic, to make sure either the old or the new history service stack will be used, but not both. However, I don't cover mis config of within the scope of the old history service stack itself. For example, ApplicationHistoryStore -> null store while enabling history service. It even didn't work in the previous situation. bq. I mean to define "hadoop.tmp.dir" in YarnConfiguration to be something like: HADOOP_TMP_DIR which sounds more uniform when dealing with config. "hadoop.tmp.dir" should be part of YarnConfiguration. If it really needs to be added, it should be placed in CommonConfigurationKeys. However, I'm afraid it's not a good idea to do that, either. Let's look into the default of it. {code} hadoop.tmp.dir /tmp/hadoop-${user.name} A base for other temporary directories. {code} The default comes with a param, which can not be determined upfront either. AFAIK, all such kind of defaults are not contained in config classes. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100623#comment-14100623 ] Junping Du commented on YARN-2033: -- bq. Because I want to check whether the application exists in the timeline store or not, before retrieving the application attempt information. If the application doesn't exist, we need to throw ApplicationNotFoundException. IMO, this is not necessary as application should exist in most cases and we don't need to duplicated visit LevelDB twice. If application doesn't exist, we can throw ApplicationNotFoundException in retrieving app attempt info. Isn't it? bq. I rethink about the backward compatibility, and I think it's not good to reply on checking APPLICATION_HISTORY_STORE, because its default is already the FS-based history store. The users may use this store without explicitly setting it in their config file. Instead, I think it's more reasonable to check APPLICATION_HISTORY_ENABLED to determine whether the user is using old history store, because it is false by default. Backward compatibility is only one concern I had. Another concern here is on usability of these (old and new) configurations. I just list one possible wrong configuration above but didn't want to judge which wrong configuration is more likely to happen. The point here is we should check on the combination of related configurations and make all wrong combinations get warned. Any concern on doing this? bq. This is because conf.get("hadoop.tmp.dir") cannot be determined in advance. I mean to define "hadoop.tmp.dir" in YarnConfiguration to be something like: HADOOP_TMP_DIR which sounds more uniform when dealing with config. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100450#comment-14100450 ] Hadoop QA commented on YARN-2033: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12662437/YARN-2033.6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 17 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationMasterServiceOnHA org.apache.hadoop.yarn.client.TestRMFailover org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4660//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4660//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, > YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100356#comment-14100356 ] Zhijie Shen commented on YARN-2033: --- [~djp], thanks for your comments. I addressed most of your comments in the new patch, and fix one bug I've found locally. Bellow are some response w.r.t to your concerns. bq. Why do we need getApplication(appAttemptId.getApplicationId(), ApplicationReportField.NONE) here? Because I want to check whether the application exists in the timeline store or not, before retrieving the application attempt information. If the application doesn't exist, we need to throw ApplicationNotFoundException. BTW, in YARN-1250, getting app is going to be required for each API, because we need to check whether the user has access to this application or not. bq. If user's config is slightly wrong (let's assume: YarnConfiguration.APPLICATION_HISTORY_STORE != null, YarnConfiguration.RM_METRICS_PUBLISHER_ENABLED=true), then here we disable yarnMetricsEnabled sliently which make trouble-shooting effort a little harder. Suggest to log warn messages when user's wrong configuration happens. Better to move logic operations inside of if() to a separated method and log the error for wrong configuration. I rethink about the backward compatibility, and I think it's not good to reply on checking APPLICATION_HISTORY_STORE, because its default is already the FS-based history store. The users may use this store without explicitly setting it in their config file. Instead, I think it's more reasonable to check APPLICATION_HISTORY_ENABLED to determine whether the user is using old history store, because it is false by default. Unless the user sets it explicitly in the config file, he's not able to use the old history store. Therefore I changed the logic in YarnClientImpl, ApplicationHistoryServer, YarnMetricsPublisher to reply on APPLICATION_HISTORY_ENABLED for backward compatibility. Per the suggestion, if if the old history service stack is used, a warn level log will be recorded. In addition, when APPLICATION_HISTORY_ENABLED = true, YarnMetricsPublisher cannot be enabled, preventing RMApplicationHistoryWriter and YarnMetricsPublisher writing the application history simultaneously. bq. The method of convertToApplicationReport seems a little too sophisticated in creating applicationReport. Another option is to wrapper it as Builder pattern (plz refer in MiniDFSCluster) should be better. I agree the builder model should be more decent, but it seems that it needs to change Report classes, which currently use newInstance to construct the instance. Let's file a separate Jira to deal with building a big record with quite a few fields. bq. We should replace "hadoop.tmp.dir" and "/yarn/timeline/generic-history" with constant string in YarnConfiguration. BTW, "hadoop.tmp.dir" may not be necessary? This is because conf.get("hadoop.tmp.dir") cannot be determined in advance. bq. For public API (although marked as unstable), adding a new exception will break compatibility of RPC as old version client don't know how to deal with new exception. ApplicationContext is actually not an RPC interface, but is used internally in the server daemons. We previously refactored the code and created such common interface for RM and GHS to source the application/attempt/container report(s) (although RM still pulls the information from RMContext directly, such that we could use the same CLI/webUI/service, but hook on different data source. Anyway, the annotations here are misleading, such that I deleted them. bq. I am not sure if this change (and other changes in this class) is necessary. If not, we can remove it. I did this intentionally. In fact, I wanted to discard {code} protected int allocatedMB; protected int allocatedVCores; {code} Because history information doesn't include the runtime resource usage information. If we keep the two fields here, in the web services output, we will always see allocatedMB=0, and allocatedVCores=0. bq. We already have the same implementation of MultiThreadedDispatcher in RMApplicationHistoryWriter.java. That's right. Again it's duplicated by purpose. After this patch, I'm going to deprecate the classes of the old generic history read/write layer, including RMApplicationHistoryWriter (YARN-2320), such that in the next big release (e.g. Hadoop 3.0), we can remove the deprecated code. MultiThreadedDispatcher should be the sub-component of YarnMetricsPublisher unless it is going be used by other components. It it happens, we can promote it to the first-citizen class. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN >
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099974#comment-14099974 ] Junping Du commented on YARN-2033: -- [~zjshen], thanks for comments above which sounds good to me. I just go through your latest patch, a couple of comments: {code} + public static final String RM_METRICS_PUBLISHER_MULTI_THREADED_DISPATCHER_POOL_SIZE = + RM_PREFIX + "metrics-publisher.multi-threaded-dispatcher.pool-size"; + public static final int DEFAULT_RM_METRICS_PUBLISHER_MULTI_THREADED_DISPATCHER_POOL_SIZE = + 10; {code} The name of config looks like too long. May be we can rename it to something shorter, i.e. RM_PREFIX + "metrics-publisher.dispatcher.pool-size"? {code} - optional string diagnostics = 5 [default = "N/A"]; - optional YarnApplicationAttemptStateProto yarn_application_attempt_state = 6; - optional ContainerIdProto am_container_id = 7; + optional string original_tracking_url = 5; + optional string diagnostics = 6 [default = "N/A"]; + optional YarnApplicationAttemptStateProto yarn_application_attempt_state = 7; + optional ContainerIdProto am_container_id = 8; {code} We shouldn't insert a new field as it will change the order of existing fields. In PB, the encoded messages only include field type and number and will be map to field name when decoding. Thus, change the field number here will break compatibility which is unnecessary. Add original_tracking_url with field number of 8 should be fine. {code} -if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, - YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) { - historyServiceEnabled = true; +if (conf.get(YarnConfiguration.APPLICATION_HISTORY_STORE) == null +&& conf.getBoolean(YarnConfiguration.RM_METRICS_PUBLISHER_ENABLED, +YarnConfiguration.DEFAULT_RM_METRICS_PUBLISHER_ENABLED) || +conf.get(YarnConfiguration.APPLICATION_HISTORY_STORE) != null +&& conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, +YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) { + yarnMetricsEnabled = true; {code} If user's config is slightly wrong (let's assume: YarnConfiguration.APPLICATION_HISTORY_STORE != null, YarnConfiguration.RM_METRICS_PUBLISHER_ENABLED=true), then here we disable yarnMetricsEnabled sliently which make trouble-shooting effort a little harder. Suggest to log warn messages when user's wrong configuration happens. Better to move logic operations inside of if() to a separated method and log the error for wrong configuration. {code} + + The setting that controls whether yarn metrics is published on +the timeline server or not by RM. + yarn.resourcemanager.metrics-publisher.enabled + false + {code} Indentation should be 2 white spaces instead of tab. In ApplicationHistoryManagerOnTimelineStore.java, {code} } catch (YarnException e) { + throw new IOException(e); +} {code} This kind of exception translate is unnecessary to me. We can remove it as YarnException get throw here. If we decide to throw IOException only (please see my comments later), we can extend the block to cover more code that could throw YarnException and translate to IOException. The method of convertToApplicationReport seems a little too sophisticated in creating applicationReport. Another option is to wrapper it as Builder pattern (plz refer in MiniDFSCluster) should be better. The same comments on convertToApplicationAttemptReport and convertToContainerReport. This is only optional comments, see if you want to address here or some separate JIRA in future. {code} + public ApplicationAttemptReport getApplicationAttempt( + ApplicationAttemptId appAttemptId) throws YarnException, IOException { +getApplication(appAttemptId.getApplicationId(), ApplicationReportField.NONE); +TimelineEntity entity = null; ... {code} Why do we need getApplication(appAttemptId.getApplicationId(), ApplicationReportField.NONE) here? IMO, the only work here is to check if applicationId is valid, but we have check on appAttemptId later. so we may consider to remove it if unnecessary. In addition, may be ApplicationReportField.NONE is not useful? {code} -new Path(conf.get(YarnConfiguration.FS_APPLICATION_HISTORY_STORE_URI)); +new Path(conf.get(YarnConfiguration.FS_APPLICATION_HISTORY_STORE_URI, +conf.get("hadoop.tmp.dir") + "/yarn/timeline/generic-history")); {code} We should replace "hadoop.tmp.dir" and "/yarn/timeline/generic-history" with constant string in YarnConfiguration. BTW, "hadoop.tmp.dir" may not be necessary? In ApplicationContext.java, {code} * @return {@link ApplicationReport} for the ApplicationId. + * @throws YarnException * @throws IOException */ @Public @Unstable - ApplicationReport getApplication(ApplicationId appId) throws IOExcep
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096592#comment-14096592 ] Zhijie Shen commented on YARN-2033: --- [~djp], thanks for raising this question explicitly. Here're two points I'd like to highlight for this work: 1. This patch doesn't intend to remove the existing FS based history store, but deprecate it by removing the default configs about loading FS based history store. On the other hand, the patch adds the history store that rides the timeline store, and use it as the default. Given the user who is the early adopter of the generic history service wants to continue with FS based history store, he needs to set the old configs explicitly (actually he should have done it because by default the generic history service is not enabled), and the new generic history service is still going to horner old configs for backward compatibility. 2. Though the generic history service (previously we call it application history server) is introduced to Hadoop since 2.4, but it is not production ready. We have explicitly highlighted it in the [documentation|http://hadoop.apache.org/docs/r2.4.0/hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Current_Status]. I agree it seems to be a bit aggressive to move from FS based history store to timeline store based one as the default, however, I'm afraid it's the best choice at the current stage, because FS based history store has several critical limitations: no caching, no retention, not scalable and not supporting the secure mode. Unless we're able to solve all these problems (obviously we don't have the bandwidth to do it now), it's risky to use FS based history store as the default, in particular when the timeline server is going to be production ready. On the other side, the aforementioned limitations have already been addressed by the timeline store (scalability will be ensured by HBase timeline store). Hence timeline store based history store should be a more reasonable and reliable default of new users. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096535#comment-14096535 ] Junping Du commented on YARN-2033: -- Thanks [~zjshen] for updating the patch. I am reviewing it now and a major concern here is shall we replace GHS to base on TimelineStore by default? Or we should keep the original localFS implementation by default. Previous one sounds a little aggressive to me, especially for a new feature. Thoughts? > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093742#comment-14093742 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12661126/YARN-2033.5.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 16 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4599//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4599//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.5.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, > YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087066#comment-14087066 ] Junping Du commented on YARN-2033: -- bq. I prefer not to do it right now, because moving the minor change out is not going to squeeze this patch significant. Instead, I've to maintain a longer dependency chain of the outstanding patches. Do you agree? Agree. I will review your sub-jira first. After that get committed, then we can come back and hope changes can be smaller. Thanks for reminding the sub JIRAs. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084869#comment-14084869 ] Zhijie Shen commented on YARN-2033: --- [~djp], thanks for you volunteering to review this big patch. bq. I just take a glance at latest patch which looks pretty big for me and mix of code refactor (move location of TimelineClient, etc.), In case of the big refactor work, I've already had two jiras for them: YARN-2298 and YARN-2302. Please note that in this jira I posted the patch of generic-history in timeline store only and the uber patch including all code changes (with _ALL suffix). bq. side logic like: adding originalURL to ApplicationAttemptReport, etc. Can we break this patch to several pieces and create a sub JIRA for each one? I prefer not to do it right now, because moving the minor change out is not going to squeeze this patch significant. Instead, I've to maintain a longer dependency chain of the outstanding patches. Do you agree? > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084670#comment-14084670 ] Junping Du commented on YARN-2033: -- Thanks [~zjshen] for uploading a proposal and keep updating patches! The proposal sounds like a good direction. I just take a glance at latest patch which looks pretty big for me and mix of code refactor (move location of TimelineClient, etc.), side logic like: adding originalURL to ApplicationAttemptReport, etc. Can we break this patch to several pieces and create a sub JIRA for each one? That should be much easier for review. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081227#comment-14081227 ] Hadoop QA commented on YARN-2033: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658949/YARN-2033_ALL.4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 20 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4498//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4498//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068546#comment-14068546 ] Zhijie Shen commented on YARN-2033: --- The test failures are not related. See YARN-2304. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068397#comment-14068397 ] Hadoop QA commented on YARN-2033: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656841/YARN-2033_ALL.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 20 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestApplicationMasterServiceOnHA org.apache.hadoop.yarn.util.TestFSDownload org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4381//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4381//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, > YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, > YARN-2033_ALL.3.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068217#comment-14068217 ] Hadoop QA commented on YARN-2033: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656819/YARN-2033_ALL.2.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4378//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.Prototype.patch, > YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066246#comment-14066246 ] Hadoop QA commented on YARN-2033: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656482/YARN-2033_ALL.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 20 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.util.TestFSDownload org.apache.hadoop.yarn.server.resourcemanager.metrics.TestYarnMetricsPublisher {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4358//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-YARN-Build/4358//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/4358//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/4358//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4358//console This message is automatically generated. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.1.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061889#comment-14061889 ] Zhijie Shen commented on YARN-2033: --- bq. I would think the timeline store might have to supporting storing a lot more information than the history store. In that case, one might want to keep them separate? IMHO, it depends on the use cases. Either generic or the application specific metrics can be a lot, or both. The problem to keep them separately is two have two different set of interfaces, which double our effort of maintenance and upgrade. For example, we've done data retention, and caching for the Leveldb-based timeline store. However, generic history cannot taken advantage of it unless we implement the similar features for the application history store again. I don't think keeping both metrics sets into the same store will be a big trouble for each other. With the uniformed the store interface, we can restrict the effort of improving the store implementation, where can isolate the two metrics sets (e.g., storing in two tables). > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, > YARN-2033.Prototype.patch > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061873#comment-14061873 ] Zhijie Shen commented on YARN-2033: --- Reassign the ticket to myself. > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf > > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995438#comment-13995438 ] Karthik Kambatla commented on YARN-2033: Thanks Vinod. Would like to hear your thoughts on the following: What are the perceived scalability requirements of the history store and timeline store? I would think the timeline store might have to supporting storing a lot more information than the history store. In that case, one might want to keep them separate? > Investigate merging generic-history into the Timeline Store > --- > > Key: YARN-2033 > URL: https://issues.apache.org/jira/browse/YARN-2033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > > Having two different stores isn't amicable to generic insights on what's > happening with applications. This is to investigate porting generic-history > into the Timeline Store. > One goal is to try and retain most of the client side interfaces as close to > what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)