[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130292#comment-14130292 ] Junping Du commented on YARN-2033: -- Seems not work. [~zjshen], can you provide a new one here? Thanks! Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, YARN-2033.12.patch, YARN-2033.13.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130479#comment-14130479
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668114/YARN-2033.13.patch
against trunk revision bf64fce.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4893//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4893//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch,
YARN-2033.12.patch, YARN-2033.13.patch, YARN-2033.13.patch,
YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch,
YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, YARN-2033.9.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128877#comment-14128877 ] Vinod Kumar Vavilapalli commented on YARN-2033: --- Looks mostly good. Rename yarn.resourcemanager.metrics-publisher.enabled - also to say system-metrics-publisher too? Similarly rename yarn.resourcemanager.metrics-publisher.dispatcher.pool-size? Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129042#comment-14129042
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667818/YARN-2033.12.patch
against trunk revision 47bdfa0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4874//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4874//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch,
YARN-2033.12.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch,
YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch,
YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129354#comment-14129354 ] Junping Du commented on YARN-2033: -- Sure. Will commit it soon. Thanks [~zjshen] for the patch and [~vinodkv] for review! Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.11.patch, YARN-2033.12.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127384#comment-14127384
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667445/YARN-2033.10.patch
against trunk revision 90c8ece.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4858//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4858//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.10.patch, YARN-2033.2.patch, YARN-2033.3.patch,
YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch,
YARN-2033.8.patch, YARN-2033.9.patch, YARN-2033.Prototype.patch,
YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch,
YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126073#comment-14126073
]
Vinod Kumar Vavilapalli commented on YARN-2033:
---
Mostly looks fine, this is a rapidly changing part of the code-base! I get a
feeling we need some umbrella cleanup effort to make consistent usage w.r.t
history-service/timeline-service. Anyways, some comments
- RMApplicationHistoryWriter is not really needed anymore. We did document it
to be unstable/alpha too. We can remove it directly instead of deprecating it.
It's a burden to support two interface hierarchies. I'm okay doing it
separately though.
- YarnClientImpl: Calls using AHSClient shouldn't rely on timeline-publisher
yet, we should continue to use APPLICATION_HISTORY_ENABLED for that till we get
rid of AHSClient altogether. We should file a ticket for this too.
- You removed the unstable annotations from ApplicationContext APIs. We should
retain them, this stuff isn't stable yet.
- Rename YarnMetricsPublisher - {Platform|System}MetricsPublisher to avoid
confusing it with host/daemon metrics that exist outside today?
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126236#comment-14126236
]
Zhijie Shen commented on YARN-2033:
---
[~vinodkv], thanks for your comments. I've updated the patch accordingly.
bq. RMApplicationHistoryWriter is not really needed anymore. We did document it
to be unstable/alpha too. We can remove it directly instead of deprecating it.
It's a burden to support two interface hierarchies. I'm okay doing it
separately though.
Seems to make sense. Previously I created a ticket for deprecating the old
history store stack. Let me update that jira.
bq. YarnClientImpl: Calls using AHSClient shouldn't rely on timeline-publisher
yet, we should continue to use APPLICATION_HISTORY_ENABLED for that till we get
rid of AHSClient altogether. We should file a ticket for this too.
In the newer patch, I revert the change in YarnClientImpl, making it use
APPLICATION_HISTORY_ENABLED. And ApplicationHistoryServer checks
APPLICATION_HISTORY_STORE for backward compatibility. This can be simplified
once the old history store stack is removed. Also I simplify the configuration
check in SystemMetricsPublisher. I'll create a jira for getting rid of
AHSClient.
bq. You removed the unstable annotations from ApplicationContext APIs. We
should retain them, this stuff isn't stable yet.
ApplicationContext is for internal usage only, not user-faced interface. So I
think it should be removed not to confuse people.
bq. Rename YarnMetricsPublisher - {Platform|System}
MetricsPublisher to avoid confusing it with host/daemon metrics that exist
outside today?
Renamed all yarnmetrics - systemmetrics.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126340#comment-14126340
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667269/YARN-2033.8.patch
against trunk revision d989ac0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4851//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4851//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126481#comment-14126481
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667301/YARN-2033.9.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4852//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4852//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.8.patch,
YARN-2033.9.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch,
YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118563#comment-14118563 ] Vinod Kumar Vavilapalli commented on YARN-2033: --- Having a quick look at this.. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117458#comment-14117458 ] Junping Du commented on YARN-2033: -- +1. Latest patch LGTM. Will commit it tomorrow if no new comments from others. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116820#comment-14116820
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665205/YARN-2033.7.patch
against trunk revision 258c7d0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4793//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4793//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114913#comment-14114913
]
Zhijie Shen commented on YARN-2033:
---
The test failure should not be related. It seems that the configuration
resource was not read correctly on jenkins:
{code}
java.lang.RuntimeException: java.util.zip.ZipException: oversubscribed dynamic
bit lengths tree
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:147)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at
org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown
Source)
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown
Source)
at
org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:153)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2334)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2322)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2393)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2346)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2263)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
at
org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:605)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247)
at
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:296)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.setup(TestRMRestart.java:119)
{code}
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114876#comment-14114876
]
Hadoop QA commented on YARN-2033:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665205/YARN-2033.7.patch
against trunk revision d8774cc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4765//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4765//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.7.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111015#comment-14111015 ] Junping Du commented on YARN-2033: -- [~zjshen], the patch looks good to me in over all. I don't have much more comments, except below. Would you fix it? bq. First of all, two queries are not duplicate: one to read the application entity, and the other to read the app attempt entity, and we previously distinguish ApplicationNotFoundException and ApplicationAttemptNotFoundException. It is always possible that App1 exists in the store with the only attempt AppAttempt1 while the user looks up for AppAttempt2. In this case, we know App1 is there, but AppAttempt2 isn't, so we will throw ApplicationAttemptNotFoundException. If we really want to differ the two exceptions, we still can check ApplicationAttempt first and check Application later (to see if throw ApplicationNotFoundException instead) if ApplicationAttemptNotFoundException get thrown there. This is more efficient as we only need to visit DB one time in most cases. Isn't it? Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100356#comment-14100356
]
Zhijie Shen commented on YARN-2033:
---
[~djp], thanks for your comments. I addressed most of your comments in the new
patch, and fix one bug I've found locally. Bellow are some response w.r.t to
your concerns.
bq. Why do we need getApplication(appAttemptId.getApplicationId(),
ApplicationReportField.NONE) here?
Because I want to check whether the application exists in the timeline store or
not, before retrieving the application attempt information. If the application
doesn't exist, we need to throw ApplicationNotFoundException.
BTW, in YARN-1250, getting app is going to be required for each API, because we
need to check whether the user has access to this application or not.
bq. If user's config is slightly wrong (let's assume:
YarnConfiguration.APPLICATION_HISTORY_STORE != null,
YarnConfiguration.RM_METRICS_PUBLISHER_ENABLED=true), then here we disable
yarnMetricsEnabled sliently which make trouble-shooting effort a little harder.
Suggest to log warn messages when user's wrong configuration happens. Better to
move logic operations inside of if() to a separated method and log the error
for wrong configuration.
I rethink about the backward compatibility, and I think it's not good to reply
on checking APPLICATION_HISTORY_STORE, because its default is already the
FS-based history store. The users may use this store without explicitly setting
it in their config file. Instead, I think it's more reasonable to check
APPLICATION_HISTORY_ENABLED to determine whether the user is using old history
store, because it is false by default. Unless the user sets it explicitly in
the config file, he's not able to use the old history store. Therefore I
changed the logic in YarnClientImpl, ApplicationHistoryServer,
YarnMetricsPublisher to reply on APPLICATION_HISTORY_ENABLED for backward
compatibility.
Per the suggestion, if if the old history service stack is used, a warn level
log will be recorded. In addition, when APPLICATION_HISTORY_ENABLED = true,
YarnMetricsPublisher cannot be enabled, preventing RMApplicationHistoryWriter
and YarnMetricsPublisher writing the application history simultaneously.
bq. The method of convertToApplicationReport seems a little too sophisticated
in creating applicationReport. Another option is to wrapper it as Builder
pattern (plz refer in MiniDFSCluster) should be better.
I agree the builder model should be more decent, but it seems that it needs to
change Report classes, which currently use newInstance to construct the
instance. Let's file a separate Jira to deal with building a big record with
quite a few fields.
bq. We should replace hadoop.tmp.dir and /yarn/timeline/generic-history
with constant string in YarnConfiguration. BTW, hadoop.tmp.dir may not be
necessary?
This is because conf.get(hadoop.tmp.dir) cannot be determined in advance.
bq. For public API (although marked as unstable), adding a new exception will
break compatibility of RPC as old version client don't know how to deal with
new exception.
ApplicationContext is actually not an RPC interface, but is used internally in
the server daemons. We previously refactored the code and created such common
interface for RM and GHS to source the application/attempt/container report(s)
(although RM still pulls the information from RMContext directly, such that we
could use the same CLI/webUI/service, but hook on different data source.
Anyway, the annotations here are misleading, such that I deleted them.
bq. I am not sure if this change (and other changes in this class) is
necessary. If not, we can remove it.
I did this intentionally. In fact, I wanted to discard
{code}
protected int allocatedMB;
protected int allocatedVCores;
{code}
Because history information doesn't include the runtime resource usage
information. If we keep the two fields here, in the web services output, we
will always see allocatedMB=0, and allocatedVCores=0.
bq. We already have the same implementation of MultiThreadedDispatcher in
RMApplicationHistoryWriter.java.
That's right. Again it's duplicated by purpose. After this patch, I'm going to
deprecate the classes of the old generic history read/write layer, including
RMApplicationHistoryWriter (YARN-2320), such that in the next big release (e.g.
Hadoop 3.0), we can remove the deprecated code. MultiThreadedDispatcher should
be the sub-component of YarnMetricsPublisher unless it is going be used by
other components. It it happens, we can promote it to the first-citizen class.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100450#comment-14100450
]
Hadoop QA commented on YARN-2033:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662437/YARN-2033.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationMasterServiceOnHA
org.apache.hadoop.yarn.client.TestRMFailover
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4660//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4660//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch,
YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch,
YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100623#comment-14100623 ] Junping Du commented on YARN-2033: -- bq. Because I want to check whether the application exists in the timeline store or not, before retrieving the application attempt information. If the application doesn't exist, we need to throw ApplicationNotFoundException. IMO, this is not necessary as application should exist in most cases and we don't need to duplicated visit LevelDB twice. If application doesn't exist, we can throw ApplicationNotFoundException in retrieving app attempt info. Isn't it? bq. I rethink about the backward compatibility, and I think it's not good to reply on checking APPLICATION_HISTORY_STORE, because its default is already the FS-based history store. The users may use this store without explicitly setting it in their config file. Instead, I think it's more reasonable to check APPLICATION_HISTORY_ENABLED to determine whether the user is using old history store, because it is false by default. Backward compatibility is only one concern I had. Another concern here is on usability of these (old and new) configurations. I just list one possible wrong configuration above but didn't want to judge which wrong configuration is more likely to happen. The point here is we should check on the combination of related configurations and make all wrong combinations get warned. Any concern on doing this? bq. This is because conf.get(hadoop.tmp.dir) cannot be determined in advance. I mean to define hadoop.tmp.dir in YarnConfiguration to be something like: HADOOP_TMP_DIR which sounds more uniform when dealing with config. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100873#comment-14100873
]
Zhijie Shen commented on YARN-2033:
---
The test failures seem to be related to RM HA.
bq. IMO, this is not necessary as application should exist in most cases and we
don't need to duplicated visit LevelDB twice. If application doesn't exist, we
can throw ApplicationNotFoundException in retrieving app attempt info. Isn't it?
First of all, two queries are not duplicate: one to read the application
entity, and the other to read the app attempt entity, and we previously
distinguish ApplicationNotFoundException and
ApplicationAttemptNotFoundException. It is always possible that App1 exists in
the store with the only attempt AppAttempt1 while the user looks up for
AppAttempt2. In this case, we know App1 is there, but AppAttempt2 isn't, so we
will throw ApplicationAttemptNotFoundException.
Moreover, when we go on with generic history ACLs, we will anyway visit the app
entity once to pull the user info for access check.
bq. The point here is we should check on the combination of related
configurations and make all wrong combinations get warned. Any concern on doing
this?
Right, so in the new patch, I've enhanced the configuration check logic, to
make sure either the old or the new history service stack will be used, but not
both. However, I don't cover mis config of within the scope of the old history
service stack itself. For example, ApplicationHistoryStore - null store while
enabling history service. It even didn't work in the previous situation.
bq. I mean to define hadoop.tmp.dir in YarnConfiguration to be something
like: HADOOP_TMP_DIR which sounds more uniform when dealing with config.
hadoop.tmp.dir should be part of YarnConfiguration. If it really needs to be
added, it should be placed in CommonConfigurationKeys. However, I'm afraid it's
not a good idea to do that, either. Let's look into the default of it.
{code}
property
namehadoop.tmp.dir/name
value/tmp/hadoop-${user.name}/value
descriptionA base for other temporary directories./description
/property
{code}
The default comes with a param, which can not be determined upfront either.
AFAIK, all such kind of defaults are not contained in config classes.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.6.patch, YARN-2033.Prototype.patch,
YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch,
YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099974#comment-14099974
]
Junping Du commented on YARN-2033:
--
[~zjshen], thanks for comments above which sounds good to me. I just go through
your latest patch, a couple of comments:
{code}
+ public static final String
RM_METRICS_PUBLISHER_MULTI_THREADED_DISPATCHER_POOL_SIZE =
+ RM_PREFIX + metrics-publisher.multi-threaded-dispatcher.pool-size;
+ public static final int
DEFAULT_RM_METRICS_PUBLISHER_MULTI_THREADED_DISPATCHER_POOL_SIZE =
+ 10;
{code}
The name of config looks like too long. May be we can rename it to something
shorter, i.e. RM_PREFIX + metrics-publisher.dispatcher.pool-size?
{code}
- optional string diagnostics = 5 [default = N/A];
- optional YarnApplicationAttemptStateProto yarn_application_attempt_state = 6;
- optional ContainerIdProto am_container_id = 7;
+ optional string original_tracking_url = 5;
+ optional string diagnostics = 6 [default = N/A];
+ optional YarnApplicationAttemptStateProto yarn_application_attempt_state = 7;
+ optional ContainerIdProto am_container_id = 8;
{code}
We shouldn't insert a new field as it will change the order of existing fields.
In PB, the encoded messages only include field type and number and will be map
to field name when decoding. Thus, change the field number here will break
compatibility which is unnecessary. Add original_tracking_url with field number
of 8 should be fine.
{code}
-if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- historyServiceEnabled = true;
+if (conf.get(YarnConfiguration.APPLICATION_HISTORY_STORE) == null
+ conf.getBoolean(YarnConfiguration.RM_METRICS_PUBLISHER_ENABLED,
+YarnConfiguration.DEFAULT_RM_METRICS_PUBLISHER_ENABLED) ||
+conf.get(YarnConfiguration.APPLICATION_HISTORY_STORE) != null
+ conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
+ yarnMetricsEnabled = true;
{code}
If user's config is slightly wrong (let's assume:
YarnConfiguration.APPLICATION_HISTORY_STORE != null,
YarnConfiguration.RM_METRICS_PUBLISHER_ENABLED=true), then here we disable
yarnMetricsEnabled sliently which make trouble-shooting effort a little harder.
Suggest to log warn messages when user's wrong configuration happens. Better to
move logic operations inside of if() to a separated method and log the error
for wrong configuration.
{code}
+ property
+ descriptionThe setting that controls whether yarn metrics is
published on
+the timeline server or not by RM./description
+ nameyarn.resourcemanager.metrics-publisher.enabled/name
+ valuefalse/value
+ /property
{code}
Indentation should be 2 white spaces instead of tab.
In ApplicationHistoryManagerOnTimelineStore.java,
{code}
} catch (YarnException e) {
+ throw new IOException(e);
+}
{code}
This kind of exception translate is unnecessary to me. We can remove it as
YarnException get throw here. If we decide to throw IOException only (please
see my comments later), we can extend the block to cover more code that could
throw YarnException and translate to IOException.
The method of convertToApplicationReport seems a little too sophisticated in
creating applicationReport. Another option is to wrapper it as Builder pattern
(plz refer in MiniDFSCluster) should be better. The same comments on
convertToApplicationAttemptReport and convertToContainerReport. This is only
optional comments, see if you want to address here or some separate JIRA in
future.
{code}
+ public ApplicationAttemptReport getApplicationAttempt(
+ ApplicationAttemptId appAttemptId) throws YarnException, IOException {
+getApplication(appAttemptId.getApplicationId(),
ApplicationReportField.NONE);
+TimelineEntity entity = null;
...
{code}
Why do we need getApplication(appAttemptId.getApplicationId(),
ApplicationReportField.NONE) here? IMO, the only work here is to check if
applicationId is valid, but we have check on appAttemptId later. so we may
consider to remove it if unnecessary. In addition, may be
ApplicationReportField.NONE is not useful?
{code}
-new Path(conf.get(YarnConfiguration.FS_APPLICATION_HISTORY_STORE_URI));
+new Path(conf.get(YarnConfiguration.FS_APPLICATION_HISTORY_STORE_URI,
+conf.get(hadoop.tmp.dir) + /yarn/timeline/generic-history));
{code}
We should replace hadoop.tmp.dir and /yarn/timeline/generic-history with
constant string in YarnConfiguration. BTW, hadoop.tmp.dir may not be
necessary?
In ApplicationContext.java,
{code}
* @return {@link ApplicationReport} for the ApplicationId.
+ * @throws YarnException
* @throws IOException
*/
@Public
@Unstable
- ApplicationReport
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14096592#comment-14096592 ] Zhijie Shen commented on YARN-2033: --- [~djp], thanks for raising this question explicitly. Here're two points I'd like to highlight for this work: 1. This patch doesn't intend to remove the existing FS based history store, but deprecate it by removing the default configs about loading FS based history store. On the other hand, the patch adds the history store that rides the timeline store, and use it as the default. Given the user who is the early adopter of the generic history service wants to continue with FS based history store, he needs to set the old configs explicitly (actually he should have done it because by default the generic history service is not enabled), and the new generic history service is still going to horner old configs for backward compatibility. 2. Though the generic history service (previously we call it application history server) is introduced to Hadoop since 2.4, but it is not production ready. We have explicitly highlighted it in the [documentation|http://hadoop.apache.org/docs/r2.4.0/hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Current_Status]. I agree it seems to be a bit aggressive to move from FS based history store to timeline store based one as the default, however, I'm afraid it's the best choice at the current stage, because FS based history store has several critical limitations: no caching, no retention, not scalable and not supporting the secure mode. Unless we're able to solve all these problems (obviously we don't have the bandwidth to do it now), it's risky to use FS based history store as the default, in particular when the timeline server is going to be production ready. On the other side, the aforementioned limitations have already been addressed by the timeline store (scalability will be ensured by HBase timeline store). Hence timeline store based history store should be a more reasonable and reliable default of new users. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093742#comment-14093742
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661126/YARN-2033.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4599//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4599//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.5.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch,
YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087066#comment-14087066 ] Junping Du commented on YARN-2033: -- bq. I prefer not to do it right now, because moving the minor change out is not going to squeeze this patch significant. Instead, I've to maintain a longer dependency chain of the outstanding patches. Do you agree? Agree. I will review your sub-jira first. After that get committed, then we can come back and hope changes can be smaller. Thanks for reminding the sub JIRAs. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084670#comment-14084670 ] Junping Du commented on YARN-2033: -- Thanks [~zjshen] for uploading a proposal and keep updating patches! The proposal sounds like a good direction. I just take a glance at latest patch which looks pretty big for me and mix of code refactor (move location of TimelineClient, etc.), side logic like: adding originalURL to ApplicationAttemptReport, etc. Can we break this patch to several pieces and create a sub JIRA for each one? That should be much easier for review. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084869#comment-14084869 ] Zhijie Shen commented on YARN-2033: --- [~djp], thanks for you volunteering to review this big patch. bq. I just take a glance at latest patch which looks pretty big for me and mix of code refactor (move location of TimelineClient, etc.), In case of the big refactor work, I've already had two jiras for them: YARN-2298 and YARN-2302. Please note that in this jira I posted the patch of generic-history in timeline store only and the uber patch including all code changes (with _ALL suffix). bq. side logic like: adding originalURL to ApplicationAttemptReport, etc. Can we break this patch to several pieces and create a sub JIRA for each one? I prefer not to do it right now, because moving the minor change out is not going to squeeze this patch significant. Instead, I've to maintain a longer dependency chain of the outstanding patches. Do you agree? Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081227#comment-14081227
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658949/YARN-2033_ALL.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 20 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4498//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4498//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068397#comment-14068397
]
Hadoop QA commented on YARN-2033:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656841/YARN-2033_ALL.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 20 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestApplicationMasterServiceOnHA
org.apache.hadoop.yarn.util.TestFSDownload
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4381//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4381//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068546#comment-14068546 ] Zhijie Shen commented on YARN-2033: --- The test failures are not related. See YARN-2304. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068217#comment-14068217
]
Hadoop QA commented on YARN-2033:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656819/YARN-2033_ALL.2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4378//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.Prototype.patch,
YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066246#comment-14066246
]
Hadoop QA commented on YARN-2033:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656482/YARN-2033_ALL.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 20 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.util.TestFSDownload
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestYarnMetricsPublisher
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4358//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4358//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4358//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4358//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4358//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061873#comment-14061873 ] Zhijie Shen commented on YARN-2033: --- Reassign the ticket to myself. Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061889#comment-14061889 ] Zhijie Shen commented on YARN-2033: --- bq. I would think the timeline store might have to supporting storing a lot more information than the history store. In that case, one might want to keep them separate? IMHO, it depends on the use cases. Either generic or the application specific metrics can be a lot, or both. The problem to keep them separately is two have two different set of interfaces, which double our effort of maintenance and upgrade. For example, we've done data retention, and caching for the Leveldb-based timeline store. However, generic history cannot taken advantage of it unless we implement the similar features for the application history store again. I don't think keeping both metrics sets into the same store will be a big trouble for each other. With the uniformed the store interface, we can restrict the effort of improving the store implementation, where can isolate the two metrics sets (e.g., storing in two tables). Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.Prototype.patch Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995438#comment-13995438 ] Karthik Kambatla commented on YARN-2033: Thanks Vinod. Would like to hear your thoughts on the following: What are the perceived scalability requirements of the history store and timeline store? I would think the timeline store might have to supporting storing a lot more information than the history store. In that case, one might want to keep them separate? Investigate merging generic-history into the Timeline Store --- Key: YARN-2033 URL: https://issues.apache.org/jira/browse/YARN-2033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Having two different stores isn't amicable to generic insights on what's happening with applications. This is to investigate porting generic-history into the Timeline Store. One goal is to try and retain most of the client side interfaces as close to what we have today. -- This message was sent by Atlassian JIRA (v6.2#6252)
