[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090685#comment-14090685 ] Hudson commented on YARN-2288: -- FAILURE: Integrated in Hadoop-Yarn-trunk #638 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/638/]) YARN-2288. Made persisted data in LevelDB timeline store be versioned. Contributed by Junping Du. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616540) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Fix For: 2.6.0 Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090827#comment-14090827 ] Hudson commented on YARN-2288: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1831 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1831/]) YARN-2288. Made persisted data in LevelDB timeline store be versioned. Contributed by Junping Du. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616540) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Fix For: 2.6.0 Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090864#comment-14090864 ] Hudson commented on YARN-2288: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1857 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/]) YARN-2288. Made persisted data in LevelDB timeline store be versioned. Contributed by Junping Du. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616540) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Fix For: 2.6.0 Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1408#comment-1408
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660309/YARN-2288-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.TestLeveldbTimelineStore
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4547//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4547//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch,
YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088903#comment-14088903 ] Junping Du commented on YARN-2288: -- Jenkins still use v4 patch, kick off test manually again. Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088980#comment-14088980
]
Hadoop QA commented on YARN-2288:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660327/YARN-2288-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4548//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4548//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch,
YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089431#comment-14089431 ] Zhijie Shen commented on YARN-2288: --- +1 for the last patch. Let's settle down the code refactor for the constants after HBase timeline store is done. Will commit the patch Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089480#comment-14089480 ] Hudson commented on YARN-2288: -- FAILURE: Integrated in Hadoop-trunk-Commit #6029 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6029/]) YARN-2288. Made persisted data in LevelDB timeline store be versioned. Contributed by Junping Du. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616540) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Fix For: 2.6.0 Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087357#comment-14087357 ] Junping Du commented on YARN-2288: -- The test failure seems to be related to configuration of testbed but not be related to the patch. Kick off Jenkins test again manually. Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-2288-v2.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087380#comment-14087380
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660025/YARN-2288-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4530//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4530//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088085#comment-14088085 ] Zhijie Shen commented on YARN-2288: --- TestTimelineWebServices fails on trunk, it seems to be broken by HADOOP-10791. I'll file a separate ticket. Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-2288-v2.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088107#comment-14088107
]
Zhijie Shen commented on YARN-2288:
---
bq. If objects in store will get lost after TS restart, we don't need it. What
do you think?
I neglect the fact of being persisted. I agree on it.
bq. Do we have plan to persistent MemoryTimelineStore?
At least we're going to have a HbaseTimelineStore. CURRENT_VERSION_INFO can is
case-by-case for each impl, but TS_STORE_VERSION_KEY is going to be a common
constant across different impls. In addition, TS_STORE_VERSION_KEY -
TIMELINE_STORE_VERSION_KEY?
some other nits:
1. T - t?
{code}
+ Incompatible version for Timeline store: expecting version
{code}
2. Unnecessary change?
{code}
- @SuppressWarnings(resource)
{code}
Other than that, I think the patch is good to go.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088655#comment-14088655 ] Junping Du commented on YARN-2288: -- Thanks for review, [~zjshen]! Please see my reply below: bq. but TS_STORE_VERSION_KEY is going to be a common constant across different impls. In addition, TS_STORE_VERSION_KEY - TIMELINE_STORE_VERSION_KEY? Actually, I had a long discussion with Jason on YARN-2045 and both of us think we should keep API (include public constant) as simple as possible. This key will not be used outside of class or sub-classes, so there is no hard requirement to put it over its parent class (an interface actually), the only value to put this to parent class is one line code reuse but this is not necessary for some other sub-classes (i.e MemoryTimelineStore) and bring extra complexity to interface which is simple now. So I prefer it to stay at sub class until HBase implementation is there and we have strong feeling to share it across different impls. Thoughts? I will fix the name issue here. bq. T - t? Nice catch. Will fix it soon. bq. Unnecessary change for - @SuppressWarnings(resource)? That just fix a Javac warning. Fix it in a separated patch sounds overkill, so include a fix here. Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088657#comment-14088657 ] Junping Du commented on YARN-2288: -- Address [~zjshen]'s comments in v3 patch. Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch, YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088841#comment-14088841
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660309/YARN-2288-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
org.apache.hadoop.yarn.server.timeline.TestLeveldbTimelineStore
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4545//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4545//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch,
YARN-2288-v4.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086538#comment-14086538
]
Zhijie Shen commented on YARN-2288:
---
The last post is incomplete, and posted by trigger wrong hot key. For the last
point, please refer to
{code}
+ throw new IOException(
+Incompatible version for Timeline store: expecting version
++ getCurrentVersion() + , but loading version + loadedVersion);
{code}
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086535#comment-14086535
]
Zhijie Shen commented on YARN-2288:
---
Thanks for the patch, [~djp]! Here're some comments:
1. Should we do the similar thing for MemoryTimelineStore? Though it is used to
for testing and playing the feature, we never prevent users from choosing it,
and it is subject to changes in the future as well.
2. Change to TIMELINE_STORE_VERSION_KEY = timeline-store-version? Should we
move the constants to TimelineStore, such that it can be inherited by different
store implementation. If CURRENT_VERSION_INFO may differ for different stores,
it can be kept here.
{code}
+ private static final String TS_STORE_VERSION_KEY = ts-store-version;
+
+ private static final Version CURRENT_VERSION_INFO = Version
+ .newInstance(1, 0);
{code}
3. Log is not correct
{code}
+LOG.info(Loaded NM state version info + loadedVersion);
{code}
4. TS - timeline? And the last comment about NM is not correct here.
{code}
+ * 1) Versioning TS store: major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0
etc.
+ * 2) Any incompatible change of TS-store is a major upgrade, and any
+ *compatible change of TS-store is a minor upgrade.
+ * 3) Within a minor upgrade, say 1.1 to 1.2:
+ *overwrite the version info and proceed as normal.
+ * 4) Within a major upgrade, say 1.2 to 2.0:
+ *throw exception and indicate user to use a separate upgrade tool to
+ *upgrade NM state or remove incompatible old state.
+ */
{code}
5. The one-line wrapper method seems to be unnecessary.
{code}
+ private void storeVersion() throws IOException {
+dbStoreVersion(CURRENT_VERSION_INFO);
+ }
+
+ // Only used for test
+ @VisibleForTesting
+ void storeVersion(Version state) throws IOException {
+dbStoreVersion(state);
+ }
{code}
6. Log an fatal message here? Though the caller(Service) seems to log it as
well, but IMHO, we shouldn't rely on it for the critical message.
{code}
checkVersion
{code}
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[ https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087118#comment-14087118 ] Junping Du commented on YARN-2288: -- Thanks [~zjshen] for review and comments! bq. Should we do the similar thing for MemoryTimelineStore? Though it is used to for testing and playing the feature, we never prevent users from choosing it, and it is subject to changes in the future as well. Do we have plan to persistent MemoryTimelineStore? If not, I don't think it is necessary to have a version here as it is used to aware old format data get loaded into new code during rollup. If objects in store will get lost after TS restart, we don't need it. What do you think? bq. Should we move the constants to TimelineStore, such that it can be inherited by different store implementation. If CURRENT_VERSION_INFO may differ for different stores, it can be kept here. I don't understand what's different TimelineStore implementations could need this in future. If so, that could be possible to be different number. From my above comments, we only need this for LeveldbTimelineStore today. Let's keep it simple? We can refactor it if need it in future. bq. The one-line wrapper method seems to be unnecessary. Nice catch. Will fix it and other incorrect comments (sorry for that copy-paste effort). bq. Log an fatal message here? Nice suggestion. Will add it there. Data persistent in timelinestore should be versioned Key: YARN-2288 URL: https://issues.apache.org/jira/browse/YARN-2288 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Affects Versions: 2.4.1 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-2288.patch We have LevelDB-backed TimelineStore, it should have schema version for changes in schema in future. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087222#comment-14087222
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660025/YARN-2288-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4528//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4528//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082076#comment-14082076
]
Hadoop QA commented on YARN-2288:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659076/YARN-2288.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4505//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4505//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
