[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834722#comment-13834722 ] Hudson commented on YARN-1239: -- FAILURE: Integrated in Hadoop-Yarn-trunk #405 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/405/]) YARN-1239. Modified ResourceManager state-store implementations to start storing version numbers. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1546229) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateVersionIncompatibleException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/RMStateVersion.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/RMStateVersionPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Fix For: 2.3.0 Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834812#comment-13834812 ] Hudson commented on YARN-1239: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1622 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1622/]) YARN-1239. Modified ResourceManager state-store implementations to start storing version numbers. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1546229) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateVersionIncompatibleException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/RMStateVersion.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/RMStateVersionPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Fix For: 2.3.0 Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834824#comment-13834824 ] Hudson commented on YARN-1239: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1596 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1596/]) YARN-1239. Modified ResourceManager state-store implementations to start storing version numbers. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1546229) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateVersionIncompatibleException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/RMStateVersion.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/RMStateVersionPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Fix For: 2.3.0 Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833601#comment-13833601
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616015/YARN-1239.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2545//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2545//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2545//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch,
YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834213#comment-13834213 ] Jian He commented on YARN-1239: --- Because java doesn't allow the derived class to reference a static variable from the parent class.. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834218#comment-13834218
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616129/YARN-1239.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2551//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2551//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch,
YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch,
YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834225#comment-13834225 ] Jian He commented on YARN-1239: --- the test failure is unrelated to this patch. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834307#comment-13834307 ] Jian He commented on YARN-1239: --- Opened YARN-1454 for tracking the test failure Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834367#comment-13834367 ] Hudson commented on YARN-1239: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4805 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4805/]) YARN-1239. Modified ResourceManager state-store implementations to start storing version numbers. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1546229) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateVersionIncompatibleException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/RMStateVersion.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/RMStateVersionPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Fix For: 2.3.0 Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834383#comment-13834383 ] Tsuyoshi OZAWA commented on YARN-1239: -- Good job! I'll update a patch for YARN-1307 based on this JIRA. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Fix For: 2.3.0 Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.8.patch, YARN-1239.9.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832595#comment-13832595
]
Tsuyoshi OZAWA commented on YARN-1239:
--
[~jianhe], maybe RMStateVersionIncompatibleException file seems to be missing
in your latest patch.
One trivial point:
{code}
- optional int64 start_time = 2;
- optional ApplicationSubmissionContextProto application_submission_context =
3;
- optional string user = 4;
+ optional ApplicationSubmissionContextProto application_submission_context =
2;
+ optional string user = 3;
+ optional int64 start_time = 4;
{code}
I think we should not swap these fields unless we think it's reasonable.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch,
YARN-1239.7.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833239#comment-13833239 ] Jian He commented on YARN-1239: --- right.. added the missing file bq. I think we should not swap these fields unless we think it's reasonable. These fields were swapped in YARN-891, this is to swap them back for protocol buffer compatibility. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.7.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833275#comment-13833275
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615943/YARN-1239.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2539//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2539//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch,
YARN-1239.7.patch, YARN-1239.8.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13831780#comment-13831780
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615639/YARN-1239.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2524//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2524//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832031#comment-13832031 ] Vinod Kumar Vavilapalli commented on YARN-1239: --- Also, please put both a JIRA comment as well as java comments in the code on the upgrade procedure - it seems like things have changed a lot since my comment [here|https://issues.apache.org/jira/browse/YARN-1239?focusedCommentId=13826891page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13826891]. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832128#comment-13832128 ] Jian He commented on YARN-1239: --- Summarize the changes: - Versioning scheme: major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0 etc. - Any incompatible change of state-store is a major upgrade, and any compatible change of state-store is a minor upgrade. - If theres's no version, treat it as 1.0. - Within a minor upgrade, say 1.1 to 1.2: overwrite the version info and proceed as normal. - Within a major upgrade, say 1.2 to 2.0: throw exception and indicate user to use a separate upgrade tool to upgrade RM state. - the version node still coexists with RMAppRoot as a separate node under StateStoreRoot Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832150#comment-13832150
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615748/YARN-1239.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2526//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch,
YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832250#comment-13832250
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615763/YARN-1239.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2529//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.5.patch, YARN-1239.6.patch,
YARN-1239.7.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830031#comment-13830031 ] Tsuyoshi OZAWA commented on YARN-1239: -- [~jianhe], could you fix the findbugs warning? The test failure looks unrelated to this JIRA to me, because [some build fails to pass TestRMRestart recently|https://www.google.com/search?q=testRMDelegationTokenRestoredOnRMRestartoq=testRMDelegationTokenRestoredOnRMRestartaqs=chrome..69i57j69i60.321j0j4sourceid=chromeespv=210es_sm=91ie=UTF-8]. This should be filed as another JIRA. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828225#comment-13828225 ] Jian He commented on YARN-1239: --- Discussed with Vinod offline, in the scope of this jira, we can just check whether this is a compatible state. If it is compatible, RM can load it and proceed as normal. In case of incompatible, throw exception and indicate user to upgrade the state with a separate tool to do things like rearranging the directory structure or renaming the fields back, etc. For that, each incompatible implementation needs to write their own version parser for parsing and loading the data. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828428#comment-13828428 ] Tsuyoshi OZAWA commented on YARN-1239: -- The scope of this JIRA looks reasonable for me. I'll review your patch later. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828485#comment-13828485
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615046/YARN-1239.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2502//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2502//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2502//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827045#comment-13827045
]
Jian He commented on YARN-1239:
---
Quote from protocol buffer guide.
{code}
If you want your new buffers to be backwards-compatible, and your old buffers
to be forward-compatible – and you almost certainly do want this – then there
are some rules you need to follow. In the new version of the protocol buffer:
you must not change the tag numbers of any existing fields.
you must not add or delete any required fields.
you may delete optional or repeated fields.
you may add new optional or repeated fields but you must use fresh tag numbers
(i.e. tag numbers that were never used in this protocol buffer, not even by
deleted fields).
{code}
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13824716#comment-13824716 ] Tsuyoshi OZAWA commented on YARN-1239: -- LGTM except findbugs warning. Could you fix it? Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13824138#comment-13824138 ] Jian He commented on YARN-1239: --- Thanks [~ozawa] for the reviews. bq. DUMMY_VERSION_INFO and VERSION_INFO should be initialized as 1L, not 1 bq. These code should pass long value. Fixed. bq. We can define strings of error messages as static final in RMStateStore I think it's fine as that static final field will only be used by test case. I wanted to assert the exception type instead of exception msg, but turns out that exception is stringfied and pass into the ExitException. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13824172#comment-13824172
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614138/YARN-1239.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2465//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2465//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2465//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823148#comment-13823148
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613976/YARN-1239.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2458//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823199#comment-13823199
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613984/YARN-1239.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2460//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2460//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2460//console
This message is automatically generated.
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823198#comment-13823198
]
Tsuyoshi OZAWA commented on YARN-1239:
--
Thank you for updating a patch, Jian. These are my review comments:
1. DUMMY_VERSION_INFO and VERSION_INFO should be initialized as 1L, not 1.
{code}
+ private final static long DUMMY_VERSION_INFO = -1;
{code}
{code}
+ protected final long VERSION_INFO = 1;
{code}
2. These code should pass long value.
{code}
+int wrongVersion = 1234;
+stateStoreHelper.writeVersion(wrongVersion);
{code}
{code}
+int rightVersion = 1;
+stateStoreHelper.writeVersion(rightVersion);
{code}
3. We can define strings of error messages as static final in RMStateStore to
avoid duplicating defining these messages in test cases.
{code}
+ Assert.assertTrue(t.getMessage().contains(
+Loading an incompatible version of Resource Manager state));
+}
{code}
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13821050#comment-13821050
]
Tsuyoshi OZAWA commented on YARN-1239:
--
IMO, version number should be defined in RMStateStore class like
TaskUmbilicalProtocol. Are there any use cases to need to declare or override
the version in configuration?
We should check version of RM when ZKRMStateStore is initialised. I've created
this function for YARN-1307 as follows:
{code}
private void ensureVersion() throws Exception {
byte[] versionData = getDataWithRetries(versionInfoPath, false).getData();
if (versionData != null) {
ByteArrayInputStream versionIs = new ByteArrayInputStream(versionData);
DataInputStream versionIn = new DataInputStream(versionIs);
try {
int currentVersion = versionIn.readInt();
if (currentVersion != VERSION_INFO) {
throw new YarnRuntimeException(Incompatible version!);
}
} finally {
versionIn.close();
}
} else {
ByteArrayOutputStream versionOs = new ByteArrayOutputStream();
DataOutputStream versionOut = new DataOutputStream(versionOs);
try {
versionOut.writeInt(VERSION_INFO);
} finally {
versionOut.close();
}
}
}
{code}
Save version information in the state store
---
Key: YARN-1239
URL: https://issues.apache.org/jira/browse/YARN-1239
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
Attachments: YARN-1239.patch
When creating root dir for the first time we should write version 1. If root
dir exists then we should check that the version in the state store matches
the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13821053#comment-13821053 ] Jian He commented on YARN-1239: --- This is a dead patch, just disregard it. I will put a new patch soon. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13821060#comment-13821060 ] Tsuyoshi OZAWA commented on YARN-1239: -- [~jianhe], do you mind if I take this JIRA, because the change in this JIRA is related to changes in YARN-1307? I can port ZK-related code from YARN-1307 to this JIRA. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13821870#comment-13821870 ] Jian He commented on YARN-1239: --- Thanks for asking, I had already created a patch and can upload very soon, you can review patch, thanks. Save version information in the state store --- Key: YARN-1239 URL: https://issues.apache.org/jira/browse/YARN-1239 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Jian He Attachments: YARN-1239.patch When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
