[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-07 Thread Jian He (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651634#comment-13651634
 ] 

Jian He commented on YARN-582:
--

tested, works on branch-2

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch, 
> YARN-582.4.patch, YARN-582.5.patch, YARN-582.6.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-07 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651633#comment-13651633
 ] 

Vinod Kumar Vavilapalli commented on YARN-582:
--

Looks good to me too.

Can you please see if the same patch works for branch-2 also? We should run 
yarn tests too on that branch just in case..

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch, 
> YARN-582.4.patch, YARN-582.5.patch, YARN-582.6.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-07 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651530#comment-13651530
 ] 

Bikas Saha commented on YARN-582:
-

Looks good. +1.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch, 
> YARN-582.4.patch, YARN-582.5.patch, YARN-582.6.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-07 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651422#comment-13651422
 ] 

Hadoop QA commented on YARN-582:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12582190/YARN-582.6.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/889//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/889//console

This message is automatically generated.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch, 
> YARN-582.4.patch, YARN-582.5.patch, YARN-582.6.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-07 Thread Jian He (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651268#comment-13651268
 ] 

Jian He commented on YARN-582:
--

bq. Why not simply create it once in the RMAppAttemptImpl and re-use it like 
earlier?
The getClientToken method has been changed to get hadoop-common-token. we can 
do that, but then we need one more api in RMAppAttempt to get yarn-api-token

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch, 
> YARN-582.4.patch, YARN-582.5.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-06 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650390#comment-13650390
 ] 

Hadoop QA commented on YARN-582:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12581994/YARN-582.5.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/876//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/876//console

This message is automatically generated.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch, 
> YARN-582.4.patch, YARN-582.5.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-06 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650337#comment-13650337
 ] 

Bikas Saha commented on YARN-582:
-

Move into a common function?
{code}
  Credentials credentials = new Credentials();
  Token appToken = 
appAttempt.getApplicationToken();
  if(appToken != null){
credentials.addToken(appToken.getService(), appToken);
  }
  Token clientToken = appAttempt.getClientToken();
  if(clientToken != null){
credentials.addToken(clientToken.getService(), clientToken);
  }

  ApplicationAttemptState attemptState =
  new ApplicationAttemptState(appAttempt.getAppAttemptId(),
appAttempt.getMasterContainer(), credentials);
{code}

We should not catch and ignore the exception. Lets catch it in existing catch 
block so that attempt gets notified of error.
{code}
try {
  credentials.writeTokenStorageToStream(dob);
} catch (IOException e) {
  LOG.warn("Error converting credentials to bytebuffer" +
" while storing attempt: " + 
attemptState.getAttemptId());
}
{code}

Lets add "because running attempts are rebooted"
{code}
  // For now, no need to populate tokens back to
  // ApplicationTokenSecretManager.
{code}

Why not simply create it once in the RMAppAttemptImpl and re-use it like 
earlier?
{code}
  origTrackingUrl = this.currentAttempt.getOriginalTrackingUrl();
  Token attemptClientToken =
  this.currentAttempt.getClientToken();
  if (attemptClientToken != null) {
clientToken =
BuilderUtils.newClientToken(attemptClientToken.getIdentifier(),
  attemptClientToken.getKind().toString(), attemptClientToken
.getPassword(), attemptClientToken.getService().toString());
  }
{code}


> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch, 
> YARN-582.4.patch, YARN-582.5.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-06 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13649910#comment-13649910
 ] 

Vinod Kumar Vavilapalli commented on YARN-582:
--

bq. Alternatively, it could Credentials so that no changes are needed for 
additional tokens.
Inside the RM I'd like it to be explicit, so that we don't search in a 
credential cache each time we want a specific token.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-05 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13649449#comment-13649449
 ] 

Bikas Saha commented on YARN-582:
-

bq. RMAppAttemptImpl.appAttemptTokens should simply be 
Token
Alternatively, it could Credentials so that no changes are needed for 
additional tokens.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-02 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13648040#comment-13648040
 ] 

Bikas Saha commented on YARN-582:
-

Also, we could be better off storing credentials in rmappattemptimpl and only 
converting to bytebuffer inside rmstore. currently, because of this in 
amlauncher we end up converting bytebuffer back to credentials.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-02 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13648037#comment-13648037
 ] 

Bikas Saha commented on YARN-582:
-

Is the null check necessary? the underlying protobuf handles the null properly.
{code}
  ByteBuffer appAttemptTokens = attemptState.getAppAttemptTokens();
  if(appAttemptTokens != null){
attemptStateData.setAppAttemptTokens(appAttemptTokens);
  }
{code}

New public method necessary? RMAppAttemptImpl.recoverAppAttemptTokens()

Looks like all changes in RMAppImpl are unnecessary.

Bug in existing testDelegationTokenRestoredOnRMrestart(). The assert check 
should be made for rm1 and also for rm2. Right?
{code}
// start new RM
MockRM rm2 = new TestSecurityMockRM(conf, memStore);
rm2.start();

// verify tokens are properly populated back to DelegationTokenRenewer
Assert.assertEquals(tokenSet, rm1.getRMContext()
  .getDelegationTokenRenewer().getDelegationTokens());
{code}

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-01 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646835#comment-13646835
 ] 

Hadoop QA commented on YARN-582:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12581393/YARN-582.3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/855//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/855//console

This message is automatically generated.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-05-01 Thread Jian He (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646806#comment-13646806
 ] 

Jian He commented on YARN-582:
--

new patch addressed last comments

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch, YARN-582.3.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-04-30 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646282#comment-13646282
 ] 

Bikas Saha commented on YARN-582:
-

Why is this inside the try instead of where the existing fields are set. 
Isnt is safe to pass null to setAppAttemptTokens?
{code}
   try {
+if(appAttemptTokens != null){
+  attemptStateData.setAppAttemptTokens(appAttemptTokens);
+}
{code}

Why create new token secret manager for every generateToken() call?
{code}
+  private ByteBuffer generateTokens(ApplicationAttemptId attemptId,
+  Configuration conf) {
+ApplicationTokenSecretManager appTokenMgr =
+new ApplicationTokenSecretManager(conf);
+ApplicationTokenIdentifier appTokenId =
+new ApplicationTokenIdentifier(attemptId);
{code}

This check should be performed after restart also.
{code}
+// assert application Token is saved
+Assert.assertEquals(attempt1Token, attemptState.getAppAttemptTokens());
{code}

This check should be performed before restart also since we changed this code 
path.
{code}
+// assert ApplicationTokenSecretManager has the password populated
+Assert.assertTrue(rm2.getApplicationTokenSecretManager().hasPassword(
+  newAttempt.getAppAttemptId()));
{code}

This is wrong because the new app should be creating its own tokens.
{code}
 }
+app.createNewAttempt(true);
+break;
+  case RECOVER:
+RMAppAttempt attempt = app.createNewAttempt(true);
+
+// reuse the appToken from previous attempt
+if (UserGroupInformation.isSecurityEnabled()) {
+  ApplicationAttemptId previousAttempt =
+  Records.newRecord(ApplicationAttemptId.class);
+  previousAttempt.setApplicationId(app.getApplicationId());
+  previousAttempt.setAttemptId(app.getAppAttempts().size() - 1);
+  ApplicationState appState = app.getRMState().getApplicationState()
+  .get(app.getApplicationId());
+  ApplicationAttemptState attemptState =
+  appState.getAttempt(previousAttempt);
+  assert attemptState != null;
+  ((RMAppAttemptImpl) attempt).recoverAppAttemptTokens(attemptState
+.getAppAttemptTokens());
+}
+break;
+  default:
{code}
Hence this is wrong
{code}
+// assert the new Attempt id is the same as the desired new attempt id
+Assert.assertEquals(desiredNewAttemptId, newAttempt.getAppAttemptId());
+
+// assert new attempt reuse previous attempt tokens
+Assert.assertEquals(attempt1Token, newAttempt.getAppAttemptTokens());
{code}

Need to check for securityEnabled when recovering tokens and populating the 
secret manager?

Can we move token creation from constructor of RMAppAttemptImpl to 
AttemptStartedTransition? That way we will not end up creating new tokens in 
constructor and overriding them in recover(). Also in recover(), lets just 
populate the tokens but not add them to the secret managers. Later in work 
preserving restart we need to create a NEW->RUNNING transition in which the 
restored tokens will be added to the secret manager.

Things to check
1) Why is this code in FinalTransition and not BaseFinalTransition?
{code}
  // Unregister from the ClientTokenSecretManager
  if (UserGroupInformation.isSecurityEnabled()) {
appAttempt.rmContext.getClientToAMTokenSecretManager()
  .unRegisterApplication(appAttempt.getAppAttemptId());
  }
{code}
2) why is this duplicated in both BaseFinalTransition and 
AMUnregisteredTransition?
{code}
  // Remove the AppAttempt from the ApplicationTokenSecretManager
  appAttempt.rmContext.getApplicationTokenSecretManager()
.applicationMasterFinished(appAttemptId);
{code}

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-582) Restore appToken for app attempt after RM restart

2013-04-30 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646095#comment-13646095
 ] 

Hadoop QA commented on YARN-582:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12581243/YARN-582.2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/847//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/847//console

This message is automatically generated.

> Restore appToken for app attempt after RM restart
> -
>
> Key: YARN-582
> URL: https://issues.apache.org/jira/browse/YARN-582
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-582.1.patch, YARN-582.2.patch
>
>
> These need to be saved and restored on a per app attempt basis. This is 
> required only when work preserving restart is implemented for secure 
> clusters. In non-preserving restart app attempts are killed and so this does 
> not matter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira