[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760988#comment-13760988 ] Hudson commented on YARN-1107: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #325 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/325/]) YARN-1107. Fixed a bug in ResourceManager because of which RM in secure mode fails to restart. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1520726) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Fix For: 2.1.1-beta Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch, YARN-1107.20130906.1.patch, YARN-1107.20130906.2.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761027#comment-13761027 ] Hudson commented on YARN-1107: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1515 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1515/]) YARN-1107. Fixed a bug in ResourceManager because of which RM in secure mode fails to restart. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1520726) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Fix For: 2.1.1-beta Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch, YARN-1107.20130906.1.patch, YARN-1107.20130906.2.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761042#comment-13761042 ] Hudson commented on YARN-1107: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1541 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1541/]) YARN-1107. Fixed a bug in ResourceManager because of which RM in secure mode fails to restart. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1520726) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Fix For: 2.1.1-beta Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch, YARN-1107.20130906.1.patch, YARN-1107.20130906.2.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760757#comment-13760757 ] Omkar Vinit Joshi commented on YARN-1107: - Thanks Vinod.. bq. Put both RMDelegationTokenSecretManager and ClientRMService in RMContext. Then you don't need delegationTokenRenewer.setClientRMService() and ClientRMService.getDelegationTokenSecretManager(). bq. You can add an assert in DelegationTokenRenewer.serviceStart() to check for ClientRMService.start() after the comment. It'll be useful if tests enable assertions, can you check? done.. bq. RMDelegationTokenIdentifier.Renewer.setSecretManager is moved into ClientRMService, but not so in the test. Can we change that. fixed bq. Please also take care of the test-issue and the findbugs warning. Done.. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Fix For: 2.1.1-beta Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[
https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760826#comment-13760826
]
Hadoop QA commented on YARN-1107:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12601937/YARN-1107.20130906.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1868//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1868//console
This message is automatically generated.
Job submitted with Delegation token in secured environment causes RM to fail
during RM restart
--
Key: YARN-1107
URL: https://issues.apache.org/jira/browse/YARN-1107
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Omkar Vinit Joshi
Priority: Blocker
Fix For: 2.1.1-beta
Attachments: rm.log, YARN-1107.20130828.1.patch,
YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch,
YARN-1107.20130906.1.patch
If secure RM with recovery enabled is restarted while oozie jobs are running
rm fails to come up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760870#comment-13760870 ] Omkar Vinit Joshi commented on YARN-1107: - removing locking.. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Fix For: 2.1.1-beta Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch, YARN-1107.20130906.1.patch, YARN-1107.20130906.2.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760925#comment-13760925 ] Vinod Kumar Vavilapalli commented on YARN-1107: --- +1 for the latest patch. I just made sure that without the core change, TestRMRestart fails. So we are good to go. Checking this in. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Fix For: 2.1.1-beta Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch, YARN-1107.20130906.1.patch, YARN-1107.20130906.2.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760930#comment-13760930 ] Hudson commented on YARN-1107: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4383 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4383/]) YARN-1107. Fixed a bug in ResourceManager because of which RM in secure mode fails to restart. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1520726) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Fix For: 2.1.1-beta Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch, YARN-1107.20130906.1.patch, YARN-1107.20130906.2.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13758303#comment-13758303 ] Omkar Vinit Joshi commented on YARN-1107: - Thanks vinod.. bq. In DelegationTokenRenewer, you are breaking the following assumption, which was put in via YARN-280 yeah fixed it. bq. Leave a comment in DelegationTokenRenewer.serviceStart() as to what we are really doing w.r.t pendingTokenForRenewal. Yes added one. bq. Not just in the test-code, can you move the token-short-circuit setting from ClientRMService into DelegationTokenRenewer? fixed. moved the code from ClientRMService to DelegationTokenRenewer. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[
https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13758419#comment-13758419
]
Hadoop QA commented on YARN-1107:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12601476/YARN-1107.20130904.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1836//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1836//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1836//console
This message is automatically generated.
Job submitted with Delegation token in secured environment causes RM to fail
during RM restart
--
Key: YARN-1107
URL: https://issues.apache.org/jira/browse/YARN-1107
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Omkar Vinit Joshi
Priority: Blocker
Fix For: 2.1.1-beta
Attachments: rm.log, YARN-1107.20130828.1.patch,
YARN-1107.20130829.1.patch, YARN-1107.20130904.1.patch
If secure RM with recovery enabled is restarted while oozie jobs are running
rm fails to come up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754044#comment-13754044 ] Omkar Vinit Joshi commented on YARN-1107: - Thanks [~jianhe] for reviewing.. bq. No need to add this method 'compareRMToken' to compare Token set.The following is from JAVA Doc regarding set.equals(): Returns true if the specified object is also a set, the two sets have the same size, and every member of the specified set is contained in this set (or equivalently, every member of this set is contained in the specified set) Yes you are right but I don't want to compare masterKeyId field of the identifier as it will get renewed and updated. Hence added my custom method. bq. TestSecurityMockRM.serviceInit(): the whole method can be removed. Yes you are right.. removing it. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754059#comment-13754059 ] Jason Lowe commented on YARN-1107: -- bq. Jason Lowe and Daryn Sharp, can you confirm if this patch will work? If the RM port is ephemeral, it will be an issue. If this ends up breaking the ephemeral port case that would be unfortunate, since miniclusters run with them and it's very convenient for that use-case. I recall [~daryn] mentioning that we may need to set the bind address twice, once during serviceInit as this patch does to solve the issue and again in serviceStart once we know the actual bind address to help solve the minicluster/ephemeral port case. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754109#comment-13754109 ] Vinod Kumar Vavilapalli commented on YARN-1107: --- We can definitely set the bind-address twice, [~ojoshi] also mentioned the same. Even otherwise, we don't have secure tests with MiniYARNClusters where we validate delegation-tokens - we only have unit tests; so we should be fine. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754140#comment-13754140 ] Vinod Kumar Vavilapalli commented on YARN-1107: --- Thinking back, I am increasingly getting uncomfortable with this solution, it all seems like a hack. How about this solution (which incidentally was the first solution that occurred to me): - Start the TokenRenewer only after ClientRMService starts. - During recovery, just enqueue all the token-requests - It is TokenRenewer's responsibility to set up the short-cut. ClientRMService shouldn't have any knowledge of this. - Globally, there is a dependency of starting TokenRenewer after ClientRMService. This seems much natural to me. Shouldn't be too much of a change too. Thoughts? Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754167#comment-13754167 ] Jian He commented on YARN-1107: --- bq. I don't want to compare masterKeyId field of the identifier as it will get renewed and updated. Hence added my custom method. The masterKeyId inside the token should not be changed, even though the token is renewed. Only the DelegationKey used to generate the Token will be rolled. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754319#comment-13754319 ] Omkar Vinit Joshi commented on YARN-1107: - updating the patch. Reordering the services to make sure that RMDelegationTokenRenewer does not start before ClientRMService. Also all the tokens added for renewal will be enqueued until the service moves into STARTED state. As a part of service start it will renew all the tokens. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[
https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754331#comment-13754331
]
Hadoop QA commented on YARN-1107:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600718/YARN-1107.20130829.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1803//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1803//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1803//console
This message is automatically generated.
Job submitted with Delegation token in secured environment causes RM to fail
during RM restart
--
Key: YARN-1107
URL: https://issues.apache.org/jira/browse/YARN-1107
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Omkar Vinit Joshi
Priority: Blocker
Attachments: rm.log, YARN-1107.20130828.1.patch,
YARN-1107.20130829.1.patch
If secure RM with recovery enabled is restarted while oozie jobs are running
rm fails to come up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753232#comment-13753232 ] Omkar Vinit Joshi commented on YARN-1107: - updating title.. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[
https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753242#comment-13753242
]
Hadoop QA commented on YARN-1107:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600530/YARN-1107.20130828.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1790//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1790//console
This message is automatically generated.
Job submitted with Delegation token in secured environment causes RM to fail
during RM restart
--
Key: YARN-1107
URL: https://issues.apache.org/jira/browse/YARN-1107
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Omkar Vinit Joshi
Priority: Blocker
Attachments: rm.log, YARN-1107.20130828.1.patch
If secure RM with recovery enabled is restarted while oozie jobs are running
rm fails to come up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753278#comment-13753278 ] Jian He commented on YARN-1107: --- Patch looks fine. Few comments: No need to add this method 'compareRMToken' to compare Token set.The following is from JAVA Doc regarding set.equals(): Returns true if the specified object is also a set, the two sets have the same size, and every member of the specified set is contained in this set (or equivalently, every member of this set is contained in the specified set) So internally it should compare each member of each sets automatically and AbstractDelegationTokenIdentifier implements its own equals() already. TestSecurityMockRM.serviceInit(): the whole method can be removed. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753304#comment-13753304 ] Jian He commented on YARN-1107: --- Ran the test locally, passed with the patch and failed without. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
