[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079701#comment-14079701
]
Jason Lowe commented on YARN-2208:
--
This appears to have broken backwards compatibility with the previous release,
since the new RM cannot load an old AMRM token persisted in the state store. A
sample exception where the new RM starts with old RM state:
{noformat}
2014-07-30 11:09:17,041 FATAL [main] resourcemanager.ResourceManager
(ResourceManager.java:main(1050)) - Error starting ResourceManager
org.apache.hadoop.service.ServiceStateException: java.io.EOFException
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:837)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:877)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:874)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:874)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:918)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1047)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:87)
at
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
at
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager.addPersistedPassword(AMRMTokenSecretManager.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.recoverAppAttemptCredentials(RMAppAttemptImpl.java:740)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.recover(RMAppAttemptImpl.java:710)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recover(RMAppImpl.java:676)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:425)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1030)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:489)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
... 10 more
{noformat}
I realize this is supposed to be fixed eventually under YARN-668, but in the
interim token changes like this and YARN-2152 are routinely breaking the
ability to do upgrades without wiping the YARN state stores of the cluster.
Arguably this should either be marked as an incompatible change or the release
note should state that the RM state store needs to be wiped when upgrading.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Fix For: 2.6.0
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch,
YARN-2208.9.patch, YARN-2208.9.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079722#comment-14079722 ] Xuan Gong commented on YARN-2208: - After YARN-2211, the AMRMTokenManager state will be saved separately. We only save the currentMasterKey and NextMasterKey. And I will remove the AMRMToken from AppAttemptState. In that case, when we recover the attempt, we will not recover the AMRMToken directly from AppAttemptState. So, I think this issue for AMRMToken will be gone. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.6.0 Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079756#comment-14079756 ] Jian He commented on YARN-2208: --- [~jlowe], thanks for noting this. YARN-2212 is an immediate follow-up patch and will fix this in the same release. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.6.0 Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067475#comment-14067475 ] Hudson commented on YARN-2208: -- FAILURE: Integrated in Hadoop-Yarn-trunk #617 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/617/]) YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. Contributed by Xuan Gong (xgong: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611820) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/AMRMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestAMRMTokens.java AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.6.0 Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067524#comment-14067524 ] Hudson commented on YARN-2208: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1809 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1809/]) YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. Contributed by Xuan Gong (xgong: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611820) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/AMRMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestAMRMTokens.java AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.6.0 Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067548#comment-14067548 ] Hudson commented on YARN-2208: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1836 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1836/]) YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. Contributed by Xuan Gong (xgong: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611820) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/AMRMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestAMRMTokens.java AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.6.0 Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066559#comment-14066559 ] Jian He commented on YARN-2208: --- patch looks good AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066927#comment-14066927 ] Xuan Gong commented on YARN-2208: - Committed to trunk and branch-2. Thanks Jian for review. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.6.0 Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066943#comment-14066943 ] Hudson commented on YARN-2208: -- FAILURE: Integrated in Hadoop-trunk-Commit #5918 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5918/]) YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. Contributed by Xuan Gong (xgong: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1611820) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/AMRMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestAMRMTokens.java AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.6.0 Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.9.patch, YARN-2208.9.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065451#comment-14065451
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656318/YARN-2208.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.util.TestFSDownload
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4349//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4349//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch,
YARN-2208.9.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065525#comment-14065525
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656332/YARN-2208.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.util.TestFSDownload
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4350//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4350//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch,
YARN-2208.9.patch, YARN-2208.9.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064260#comment-14064260 ] Vinod Kumar Vavilapalli commented on YARN-2208: --- bq. Should we cache the am token password instead of re-computing password each time rpc is invoked? This is a micro-optimization. Let's get rid of caching the passwords - that way we are consistent with other token-handling in the system. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064502#comment-14064502
]
Xuan Gong commented on YARN-2208:
-
bq. This is a micro-optimization. Let's get rid of caching the passwords - that
way we are consistent with other token-handling in the system.
Removed
Also, I comment out those from
TestRMRestart::testAppAttemptTokensRestoredOnRMRestart
{code}
// TODO: fix this on YARN-2211
//TokenAMRMTokenIdentifier amrmToken = loadedAttempt1.getAMRMToken();
//Assert.assertArrayEquals(amrmToken.getPassword(),
// rm2.getRMContext().getAMRMTokenSecretManager().retrievePassword(
//amrmToken.decodeIdentifier()));
{code}
We will fix this on YARN-2211. At that ticket, we will store the currentMaster
key and nextMaster key. When RM restart happens, we can re-produce the password
based on the stored currentMaster key and nextMaster key.
But, in this ticket, since we do not cache password anymore (we only cache the
appAttemptId), and we do not have any logic to save the currentMaster key and
nextMaster key. So, those lines are expected to fail. To make a clean build, I
would like to comment them out, and put them back when we work on YARN-2211.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch, YARN-2208.8.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064511#comment-14064511
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656206/YARN-2208.8.patch
against trunk revision .
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4335//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch, YARN-2208.8.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064563#comment-14064563
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656214/YARN-2208.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.util.TestFSDownload
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4337//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4337//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064571#comment-14064571 ] Xuan Gong commented on YARN-2208: - OK. This does not sound right... Try again with the same patch. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064599#comment-14064599
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656221/YARN-2208.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.util.TestFSDownload
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4340//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4340//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch, YARN-2208.8.patch, YARN-2208.8.patch, YARN-2208.8.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059519#comment-14059519 ] Jian He commented on YARN-2208: --- patch looks good to me AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057205#comment-14057205
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654946/YARN-2208.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.util.TestFSDownload
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4254//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4254//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057219#comment-14057219 ] Xuan Gong commented on YARN-2208: - bq. org.apache.hadoop.yarn.util.TestFSDownload This is unrelated test case failure bq. org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart can pass successfully locally.. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057787#comment-14057787
]
Jian He commented on YARN-2208:
---
looks good overall,
1. Maybe change passwords to use concurrentHashMap and use read/write lock
guard nextMasterKey/currentMasterKey for better concurrency as this is a chatty
class.
2. Put In the same line
{code}
+ ms and AMRMTokenKeyActivationDelay: + this.activationDelay
+ ms);
}
else if (identifier.getKeyId() == this.currentMasterKey.getMasterKey()
{code}
3. Info level for easier debugging, while stabilizing this feature.
{code}
if (LOG.isDebugEnabled()) {
LOG.debug(Activating next master key with id:
+ this.nextMasterKey.getMasterKey().getKeyId());
}
{code}
4. createAndGetAMRMToken, add info log here also.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058069#comment-14058069 ] Xuan Gong commented on YARN-2208: - bq. 1. Maybe change passwords to use concurrentHashMap and use read/write lock guard nextMasterKey/currentMasterKey for better concurrency as this is a chatty class. DONE bq. 2. Put In the same line DONE bq. 3. Info level for easier debugging, while stabilizing this feature. DONE bq. 4. createAndGetAMRMToken, add info log here also. ADDED AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058123#comment-14058123
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655103/YARN-2208.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.util.TestFSDownload
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4263//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4263//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch,
YARN-2208.7.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058124#comment-14058124 ] Xuan Gong commented on YARN-2208: - The testcase failure : org.apache.hadoop.yarn.util.TestFSDownload is un-related AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch, YARN-2208.7.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056462#comment-14056462 ] Xuan Gong commented on YARN-2208: - bq. In which case this if condition can happen? For the test cases. Anyway, I have removed those codes. bq. merge the duplicate code DONE bq. test case: original testMasterKeyRollOver is not enough, it calls rollMasterKey to roll master key. But after this patch, the new master key will be activated only after activation delay. we need to test 1) the original token is still valid after key rolled over but before activated, and 2) invalid after the new master key is activated. 3) new token can be generated off the new master instead of old master key. DONE AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056480#comment-14056480
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654833/YARN-2208.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4237//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056540#comment-14056540
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654844/YARN-2208.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4241//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057022#comment-14057022
]
Jian He commented on YARN-2208:
---
- retriveAndSavePassword - retriveAndCachePassword
- this seems not possible to happen, given once the new token is generated, the
password should be updated as well.
{code}
// password is saved when AMRMToken is created
// When the next Master key is activated, we should
// make sure that the AMRMToken, which is created by using
// the old master key, should not work any more.
this.passwords.clear();
{code}
- assert oldKey not null
{code}
MasterKeyData oldKey = appTokenSecretManager.getMasterKey();
{code}
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057157#comment-14057157 ] Xuan Gong commented on YARN-2208: - new patch addressed all the latest comments AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch, YARN-2208.4.patch, YARN-2208.5.patch, YARN-2208.5.patch, YARN-2208.6.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055109#comment-14055109 ] Xuan Gong commented on YARN-2208: - bq. RM_AMRM_TOKEN_MASTER_KEY_ROLLING_INTERVAL_SECS - RM_AM_TOKEN_MASTER_KEY_ROLLING_INTERVAL_SECS DONE bq. RM_NMTOKEN ? DONE. Changed bq. Should we cache the am token password instead of re-computing password each time rpc is invoked? Yes, we can. Added bq. please fix the test failure also DONE AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055592#comment-14055592
]
Jian He commented on YARN-2208:
---
In which case this if condition can happen?
{code}
if (this.currentMasterKey == null this.nextMasterKey == null) {
this.currentMasterKey = createNewMasterKey();
}
{code}
merge the duplicate code
{code}
else if (identifier.getKeyId() == this.currentMasterKey.getMasterKey()
.getKeyId()) {
LOG.info(AMRMToken password retrieved successfully!!);
byte[] password = retrivePasswordInternal(identifier, currentMasterKey);
PasswordInfo pwd = new PasswordInfo(password, identifier.getKeyId());
passwords.put(applicationAttemptId, pwd);
return password;
} else if (nextMasterKey != null
identifier.getKeyId() == this.nextMasterKey.getMasterKey()
.getKeyId()) {
LOG.info(AMRMToken password retrieved successfully!!);
byte[] password = retrivePasswordInternal(identifier, nextMasterKey);
PasswordInfo pwd = new PasswordInfo(password, identifier.getKeyId());
passwords.put(applicationAttemptId, pwd);
return password;
}
{code}
test case: original testMasterKeyRollOver is not enough, it calls
rollMasterKey to roll master key. But after this patch, the new master key will
be activated only after activation delay. we need to test 1) the original token
is still valid after key rolled over but before activated, and 2) invalid after
the new master key is activated. 3) new token can be generated off the new
master instead of old master key.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch,
YARN-2208.4.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054283#comment-14054283
]
Jian He commented on YARN-2208:
---
Some comments on the patch:
- RM_AMRM_TOKEN_MASTER_KEY_ROLLING_INTERVAL_SECS -
RM_AM_TOKEN_MASTER_KEY_ROLLING_INTERVAL_SECS
- am-rm-tokens.master-key-rolling-interval-secs -
am-tokens.master-key-rolling-interval-secs
- RM_NMTOKEN ?
{code}
YarnConfiguration.RM_NMTOKEN_MASTER_KEY_ROLLING_INTERVAL_SECS
+ should be more than 2 X
+ YarnConfiguration.RM_NM_EXPIRY_INTERVAL_MS);
{code}
- Should we cache the am token password instead of re-computing password each
time rpc is invoked?
{code}
org.apache.hadoop.security.token.TokenAMRMTokenIdentifier token =
rm1.getRMContext().getRMApps().get(appAttemptId.getApplicationId())
.getRMAppAttempt(appAttemptId).getAMRMToken();
try {
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
ugi.addTokenIdentifier(token.decodeIdentifier());
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
{code}
- please fix the test failure also
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050380#comment-14050380
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653615/YARN-2208.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4179//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4179//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4179//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050572#comment-14050572
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653638/YARN-2208.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4180//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4180//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch, YARN-2208.2.patch, YARN-2208.3.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044153#comment-14044153 ] Xuan Gong commented on YARN-2208: - This patch only focused on the changes on AMRMTokenSecretManager to roll over master key periodically. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[
https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044195#comment-14044195
]
Hadoop QA commented on YARN-2208:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652512/YARN-2208.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4081//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4081//console
This message is automatically generated.
AMRMTokenManager need to have a way to roll over AMRMToken
--
Key: YARN-2208
URL: https://issues.apache.org/jira/browse/YARN-2208
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2208.1.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2208) AMRMTokenManager need to have a way to roll over AMRMToken
[ https://issues.apache.org/jira/browse/YARN-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044199#comment-14044199 ] Xuan Gong commented on YARN-2208: - The test case failure in TestRMRestart is expected. We will fix on YARN-2211. AMRMTokenManager need to have a way to roll over AMRMToken -- Key: YARN-2208 URL: https://issues.apache.org/jira/browse/YARN-2208 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-2208.1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
