[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[ https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670360#comment-13670360 ] Hudson commented on YARN-638: - Integrated in Hadoop-Mapreduce-trunk #1441 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1441/]) YARN-638. Modified ResourceManager to restore RMDelegationTokens after restarting. Contributed by Jian He. (Revision 1487720) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487720 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMDelegationTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java > Restore RMDelegationTokens after RM Restart > --- > > Key: YARN-638 > URL: https://issues.apache.org/jira/browse/YARN-638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Fix For: 2.0.5-beta > > Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch, > YARN-638.13.patch, YARN-638.14.patch, YARN-638.15.patch, YARN-638.1.patch, > YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, > YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch > > > This is missed in YARN-581. After RM restart, RMDelegationTokens need to be > added both in DelegationTokenRenewer (addressed in YARN-581), and > delegationTokenSecretManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[ https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670328#comment-13670328 ] Hudson commented on YARN-638: - Integrated in Hadoop-Hdfs-trunk #1415 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1415/]) YARN-638. Modified ResourceManager to restore RMDelegationTokens after restarting. Contributed by Jian He. (Revision 1487720) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487720 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMDelegationTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java > Restore RMDelegationTokens after RM Restart > --- > > Key: YARN-638 > URL: https://issues.apache.org/jira/browse/YARN-638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Fix For: 2.0.5-beta > > Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch, > YARN-638.13.patch, YARN-638.14.patch, YARN-638.15.patch, YARN-638.1.patch, > YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, > YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch > > > This is missed in YARN-581. After RM restart, RMDelegationTokens need to be > added both in DelegationTokenRenewer (addressed in YARN-581), and > delegationTokenSecretManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[ https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670238#comment-13670238 ] Hudson commented on YARN-638: - Integrated in Hadoop-Yarn-trunk #225 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/225/]) YARN-638. Modified ResourceManager to restore RMDelegationTokens after restarting. Contributed by Jian He. (Revision 1487720) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487720 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMDelegationTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java > Restore RMDelegationTokens after RM Restart > --- > > Key: YARN-638 > URL: https://issues.apache.org/jira/browse/YARN-638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Fix For: 2.0.5-beta > > Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch, > YARN-638.13.patch, YARN-638.14.patch, YARN-638.15.patch, YARN-638.1.patch, > YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, > YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch > > > This is missed in YARN-581. After RM restart, RMDelegationTokens need to be > added both in DelegationTokenRenewer (addressed in YARN-581), and > delegationTokenSecretManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[ https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670037#comment-13670037 ] Hudson commented on YARN-638: - Integrated in Hadoop-trunk-Commit #3808 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3808/]) YARN-638. Modified ResourceManager to restore RMDelegationTokens after restarting. Contributed by Jian He. (Revision 1487720) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487720 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/MemoryRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/NullRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMDelegationTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java > Restore RMDelegationTokens after RM Restart > --- > > Key: YARN-638 > URL: https://issues.apache.org/jira/browse/YARN-638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Fix For: 2.0.5-beta > > Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch, > YARN-638.13.patch, YARN-638.14.patch, YARN-638.15.patch, YARN-638.1.patch, > YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, > YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch > > > This is missed in YARN-581. After RM restart, RMDelegationTokens need to be > added both in DelegationTokenRenewer (addressed in YARN-581), and > delegationTokenSecretManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[ https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669975#comment-13669975 ] Vinod Kumar Vavilapalli commented on YARN-638: -- Perfect, looks good. +1. Will run tests and check it in. > Restore RMDelegationTokens after RM Restart > --- > > Key: YARN-638 > URL: https://issues.apache.org/jira/browse/YARN-638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch, > YARN-638.13.patch, YARN-638.14.patch, YARN-638.15.patch, YARN-638.1.patch, > YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, > YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch > > > This is missed in YARN-581. After RM restart, RMDelegationTokens need to be > added both in DelegationTokenRenewer (addressed in YARN-581), and > delegationTokenSecretManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669959#comment-13669959
]
Hadoop QA commented on YARN-638:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585347/YARN-638.15.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1030//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1030//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch,
> YARN-638.13.patch, YARN-638.14.patch, YARN-638.15.patch, YARN-638.1.patch,
> YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch,
> YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662701#comment-13662701
]
Hadoop QA commented on YARN-638:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583953/YARN-638.14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/969//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/969//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch,
> YARN-638.13.patch, YARN-638.14.patch, YARN-638.1.patch, YARN-638.2.patch,
> YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch,
> YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[ https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662697#comment-13662697 ] Jian He commented on YARN-638: -- bq. Also, why do we not have the corresponding records in the proto file? This will be addressed in a followup jira to avoid this patch goes too big. > Restore RMDelegationTokens after RM Restart > --- > > Key: YARN-638 > URL: https://issues.apache.org/jira/browse/YARN-638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch, > YARN-638.13.patch, YARN-638.14.patch, YARN-638.1.patch, YARN-638.2.patch, > YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch, > YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch > > > This is missed in YARN-581. After RM restart, RMDelegationTokens need to be > added both in DelegationTokenRenewer (addressed in YARN-581), and > delegationTokenSecretManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[ https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662192#comment-13662192 ] Vinod Kumar Vavilapalli commented on YARN-638: -- Also, why do we not have the corresponding records in the proto file? > Restore RMDelegationTokens after RM Restart > --- > > Key: YARN-638 > URL: https://issues.apache.org/jira/browse/YARN-638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch, > YARN-638.13.patch, YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch, > YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch, YARN-638.7.patch, > YARN-638.8.patch, YARN-638.9.patch > > > This is missed in YARN-581. After RM restart, RMDelegationTokens need to be > added both in DelegationTokenRenewer (addressed in YARN-581), and > delegationTokenSecretManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661132#comment-13661132
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583687/YARN-638.13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts
org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/950//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/950//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch,
> YARN-638.13.patch, YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch, YARN-638.7.patch,
> YARN-638.8.patch, YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13660925#comment-13660925
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583668/YARN-638.12.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/949//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.12.patch,
> YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch,
> YARN-638.5.patch, YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch,
> YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13660111#comment-13660111
]
Bikas Saha commented on YARN-638:
-
Lets keep naming consistent. Why mix *store*Token with remove*Persisted*Token.
Either use peristed or stored. Lets stick to store since existing api's on rm
state store use store.
Please open a jira to make HDFS side of the changes so that logUpdate* methods
may be removed. Please add a comment mentioning the jira in the code wherever
logUpdate* methods are called.
Are we sure that these different reads dont result in different reads to the
remote store? Why not simply read file bytes in one shot (existing code does).
Then locally work with those bytes.
{code}
+identifier.readFields(fsIn);
+long renewDate = fsIn.readLong();
{code}
Why is RMDelegationTokenManager implementing this?
{code}
+
+ @Override
+ protected void logUpdateMasterKey(DelegationKey newKey) {
+try {
+
{code}
Inconsistent logging for store and remove in RMDelegationTokenManager.
Is this definitely checking the values in the maps? This link seems to suggest
that only the keys are checked for equality.
http://docs.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object)
{code}
+Assert.assertEquals(allTokensRM1, rmDTState);
+Assert.assertEquals(allTokensRM1, allTokensRM2);
{code}
Should this be AssertEquals (same as done earlier for RM1 in the test)?
{code}
+Assert.assertTrue(rm2.getRMDTSecretManager().getAllMasterKeys()
+ .containsAll(allKeysRM1));
{code}
How about adding that old token is not there also?
{code}
+// assert new token is added into state store
+Assert.assertTrue(rmDTState.containsValue(renewDateAfterRenew));
{code}
Shouldnt the removal check for the renewed token is not in values? Also,
map.keys() should be empty, right? There should be no spurious value left
behind.
{code}
+try{
+ rm2.getRMDTSecretManager().cancelToken(token1, "user1");
+} catch(Exception e) {
+ Assert.fail();
+}
+
+// assert old token is removed from state store
+Assert.assertFalse(rmDTState.containsValue(renewDateBeforeRenew));
{code}
testRMDTMasterKeyStateOnRollingMasterKey. Instead of hand coding the token
generation and verification can we simply call rm.getDelegationToken() and then
renew the token (after key roll) to validate the same thing? Otherwise the test
will be fragile to changes in the internal code. Can we do the validation for
the restarted RM also?
What prevents the key from expiring before the token is verified? How long does
this test take to run?
testRemoveExpiredMasterKeyInRMStateStore
How do we know that expirtingKeys has 1 or more keys?
{code}
+Set expiringKeys = new HashSet();
+expiringKeys.addAll(dtSecretManager.getAllMasterKeys());
{code}
Isnt the second assert redundant? How long does this test take to run?
{code}
+// wait for expiringKeys to expire
+while(true) {
+ boolean expired = true;
+ for(DelegationKey key : expiringKeys) {
+if(rmDTMasterKeyState.contains(key)){
+ expired = false;
+}
+ }
+ if(expired)
+break;
+ Thread.sleep(500);
+}
+
+// assert initial 2 keys expired
+for (DelegationKey key : expiringKeys) {
+Assert.assertFalse(rmDTMasterKeyState.contains(key));
+}
{code}
TestRMStateStore. Why dont we put key in a set and check set equality. That
will also check that the set size == 1.
{code}
+Assert.assertEquals(key, secretManagerState.getRMDTMasterKeyState()
+ .iterator().next());
{code}
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.1.patch,
> YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch,
> YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13658989#comment-13658989
]
Hadoop QA commented on YARN-638:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583374/YARN-638.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/934//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/934//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.10.patch, YARN-638.11.patch, YARN-638.1.patch,
> YARN-638.2.patch, YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch,
> YARN-638.6.patch, YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13658705#comment-13658705
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583340/YARN-638.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.hdfs.TestClientReportBadBlock
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/933//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/933//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.10.patch, YARN-638.1.patch, YARN-638.2.patch,
> YARN-638.3.patch, YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch,
> YARN-638.7.patch, YARN-638.8.patch, YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13657442#comment-13657442
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583167/YARN-638.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/925//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/925//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/925//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch, YARN-638.7.patch,
> YARN-638.8.patch, YARN-638.9.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13656646#comment-13656646
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583039/YARN-638.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/920//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/920//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch, YARN-638.7.patch,
> YARN-638.8.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13656527#comment-13656527
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583025/YARN-638.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/919//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch, YARN-638.7.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13656207#comment-13656207
]
Bikas Saha commented on YARN-638:
-
I think the second operation will cover the first one since it will build the
tree upto the root. So we can avoid extra RPC to namenode.
{code}
fs.mkdirs(fsRootDirPath);
+fs.mkdirs(fsRMDTSecretManagerRoot);
{code}
Lets move app loading into its own function too like we did for secret manager.
{code}
+ // recover RMDTSecretManager
+ loadRMDTSecretManagerState(state);
{code}
If its not already 1 network write, can we make this 1 write operation?
{code}
+LOG.info("Storing RMDelegationToken_" + identifier.getSequenceNumber());
+identifier.write(fsOut);
+fsOut.writeLong(renewDate);
{code}
{code}
+LOG.info("Storing RMDelegationKey_" + masterKey.getKeyId());
+masterKey.write(fsOut);
+fsOut.flush();
{code}
Would be good to follow existing convention and have RMStateStore implement
storeRMDelegationToken() and abstract funtion be storeRMDelegationTokenState()
{code}
+ public abstract void storeRMDelegationToken(
+ RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate)
+ throws Exception;
+
{code}
Do we need to override renewToken() and cancelToken()? All we need to do it
have Abstract*Manager to call subclass.removeToken() at the end of
cancelToken(). similarly have renewToken() call
subclass.updateStoredToken(identifier, renewDate). We can rename
removeExpiredToken() to removeToken() and use the same method in both
cancelToken() and removeExpiredToken. Perhaps the same thing might work with
Abstract*Manager.createPassword() calling subclass.storeNewToken(identifier,
renewDate).
Much less code duplication and overriding IMO.
getAllMasterKeys() and getAllTokens() are for tests right? Then they should not
be public. If they need to be public please put @Private and @VisibleForTesting
annotations.
addPersistedRMDelegationToken() : Is there any reason why this code has been
duplicated from HDFS.DelegationTokenSecretManager() instead of moving it up to
Abstract*Manager? Similarly, addPersistedMasterKey() to Abstract*Manager and
using it HDFS/RM.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655691#comment-13655691
]
Hadoop QA commented on YARN-638:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582869/YARN-638.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/915//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/915//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch, YARN-638.6.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654681#comment-13654681
]
Bikas Saha commented on YARN-638:
-
Should HDFS code change to use this (and basically do what it was doing
earlier). Otherwise the code will still be obfuscated.
{code}
+ protected void removeMasterKey(DelegationKey key) {
+return;
+ }
{code}
Did not quite get what this is doing?
{code}
it.remove();
+if(!e.getValue().equals(currentKey))
+ removeMasterKey(e.getValue());
}
{code}
Recover secret manager before apps? Apps should be able to assume secret
manager has valid state.
{code}
rmAppManager.recover(state);
+
+// recover RMdelegationTokenSecretManager
+rmDTSecretManager.recover(state);
{code}
Can we put all secret manager state in one rmDTState object (like appState) and
then have masterKeyState and delegationTokenState inside it?
{code}
+// DTIdentifier -> renewDate
+Map rmDTState =
+new HashMap();
+
+Set rmDTMasterKeyState =
+new HashSet(
{code}
Missing javadoc for new APIs in RMStateStore
loadState()/store secret key impl is missing from filesystem store
Why store redundant rmStateStore when its already available from context?
{code}
+ protected final RMContext rmContext;
+ private RMStateStore rmStateStore;
{code}
Will this just end the current thread or actually stop the RM?
{code}
+} catch (Exception e) {
+ throw new RuntimeException("Error in removing master key", e);
+}
{code}
Dont think we should be implementing these methods. There should be explicit
store.
{code}
+ /**
+ * remove all expired master keys except current key and store the new key
+ */
+ @Override
+ protected void logUpdateMasterKey(DelegationKey newKey) {
+try {
+ rmStateStore.storeMasterKey(newKey);
+} catch (Exception e) {
+ throw new RuntimeException("Error in storing master key", e);
+}
+ }
+
+ @Override
+ protected void logExpireToken(RMDelegationTokenIdentifier ident)
+ throws IOException {
+try {
+ rmStateStore.removeRMDelegationToken(ident);
+} catch (Exception e) {
+ throw new IOException("Error in removing RMDelegationToken", e);
+}
+ }
{code}
Can just pass in a mock context that returns the test state store
{code}
+ // FOR TEST
+ public void setRMStateStore(RMStateStore rmStore) {
+this.rmStateStore = rmStore;
+ }
{code}
Is the test ensuring that the recovered tokens can be renewed by the client?
We need to carefully check the threads on which store is being called. We
cannot block the AsyncDispatcher thread.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652718#comment-13652718
]
Hadoop QA commented on YARN-638:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582417/YARN-638.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/900//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/900//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch, YARN-638.5.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652682#comment-13652682
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582410/YARN-638.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/899//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/899//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch,
> YARN-638.4.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651321#comment-13651321
]
Hadoop QA commented on YARN-638:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582162/YARN-638.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/886//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/886//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch, YARN-638.3.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-638) Restore RMDelegationTokens after RM Restart
[
https://issues.apache.org/jira/browse/YARN-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651213#comment-13651213
]
Hadoop QA commented on YARN-638:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582147/YARN-638.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/885//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/885//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/885//console
This message is automatically generated.
> Restore RMDelegationTokens after RM Restart
> ---
>
> Key: YARN-638
> URL: https://issues.apache.org/jira/browse/YARN-638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-638.1.patch, YARN-638.2.patch
>
>
> This is missed in YARN-581. After RM restart, RMDelegationTokens need to be
> added both in DelegationTokenRenewer (addressed in YARN-581), and
> delegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
