[ 
https://issues.apache.org/jira/browse/YARN-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183899#comment-14183899
 ] 

Arpit Gupta commented on YARN-2743:
-----------------------------------

Here is the stack trace from a secure run

{code}
{code}
2014-10-24 07:36:07,002 INFO  delegation.AbstractDelegationTokenSecretManager 
(AbstractDelegationTokenSecretManager.java:renewToken(452)) - Token renewal for 
identifier: owner=hrt_qa, renewer=yarn, 
realUser=oozie/ip-172-31-6-149.ec2.inter...@example.com, 
issueDate=1414136163937, maxDate=1414740963937, sequenceNumber=2, 
masterKeyId=2; total currentTokens 2
2014-10-24 07:36:07,004 WARN  security.DelegationTokenRenewer 
(DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to 
add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: RM_DELEGATION_TOKEN, Service: 
IP:8032,IP:8032, Ident: (owner=hrt_qa, renewer=yarn, 
realUser=oozie/ip-172-31-6-149.ec2.inter...@example.com, 
issueDate=1414136163937, maxDate=1414740963937, sequenceNumber=2, masterKeyId=2)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.AccessControlException: yarn is trying to 
renew a token with wrong password
        at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:476)
        at 
org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.renew(RMDelegationTokenIdentifier.java:110)
        at org.apache.hadoop.security.token.Token.renew(Token.java:377)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:474)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
        ... 6 more
2014-10-24 07:36:07,007 INFO  rmapp.RMAppImpl 
(RMAppImpl.java:rememberTargetTransitionsAndStoreState(962)) - Updating 
application application_1414136032036_0002 with final state: FAILED
2014-10-24 07:36:07,007 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(704)) - 
application_1414136032036_0002 State change from NEW to FINAL_SAVING
2014-10-24 07:36:07,008 INFO  recovery.RMStateStore 
(RMStateStore.java:transition(159)) - Updating info for app: 
application_1414136032036_0002
2014-10-24 07:36:07,062 INFO  recovery.ZKRMStateStore 
(ZKRMStateStore.java:processWatchEvent(874)) - Watcher event type: NodeCreated 
with state:SyncConnected for 
path:/rmstore/ZKRMStateRoot/RMAppRoot/application_1414136032036_0002 for 
Service org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore in 
state org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: 
STARTED
2014-10-24 07:36:07,063 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(704)) - 
application_1414136032036_0002 State change from FINAL_SAVING to FAILED
2014-10-24 07:36:07,063 WARN  capacity.CapacityScheduler 
(CapacityScheduler.java:doneApplication(769)) - Couldn't find application 
application_1414136032036_0002
2014-10-24 07:36:07,063 WARN  resourcemanager.RMAuditLogger 
(RMAuditLogger.java:logFailure(262)) - USER=hrt_qa  OPERATION=Application 
Finished - Failed TARGET=RMAppManager     RESULT=FAILURE  DESCRIPTION=App 
failed with state: FAILED       PERMISSIONS=Failed to renew token: Kind: 
RM_DELEGATION_TOKEN, Service: 172.31.6.151:8032,172.31.6.149:8032, Ident: 
(owner=hrt_qa, renewer=yarn, 
realUser=oozie/ip-172-31-6-149.ec2.inter...@example.com, 
issueDate=1414136163937, maxDate=1414740963937, sequenceNumber=2, 
masterKeyId=2)       APPID=application_1414136032036_0002
2014-10-24 07:37:08,293 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth 
successful for oozie/ip-172-31-6-149.ec2.inter...@example.com (auth:KERBEROS)
2014-10-24 07:37:08,297 INFO  authorize.ServiceAuthorizationManager 
(ServiceAuthorizationManager.java:authorize(118)) - Authorization successful 
for hrt_qa (auth:PROXY) via oozie/ip-172-31-6-149.ec2.inter...@example.com 
(auth:KERBEROS) for protocol=interface 
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
2014-10-24 07:37:08,300 INFO  delegation.AbstractDelegationTokenSecretManager 
(AbstractDelegationTokenSecretManager.java:createPassword(369)) - Creating 
password for identifier: owner=hrt_qa, renewer=yarn, 
realUser=oozie/ip-172-31-6-149.ec2.inter...@example.com, 
issueDate=1414136228299, maxDate=1414741028299, sequenceNumber=3, masterKeyId=2
2014-10-24 07:37:08,300 INFO  security.RMDelegationTokenSecretManager 
(RMDelegationTokenSecretManager.java:storeNewToken(110)) - storing RMDelegation 
token with sequence number: 3
2014-10-24 07:37:08,348 INFO  delegation.AbstractDelegationTokenSecretManager 
(AbstractDelegationTokenSecretManager.java:createPassword(369)) - Creating 
password for identifier: owner=hrt_qa, renewer=yarn, 
realUser=oozie/ip-172-31-6-149.ec2.inter...@example.com, 
issueDate=1414136228347, maxDate=1414741028347, sequenceNumber=4, masterKeyId=2
2014-10-24 07:37:08,348 INFO  security.RMDelegationTokenSecretManager 
(RMDelegationTokenSecretManager.java:storeNewToken(110)) - storing RMDelegation 
token with sequence number: 4
2014-10-24 07:37:08,611 INFO  resourcemanager.ClientRMService 
(ClientRMService.java:getNewApplicationId(281)) - Allocated new applicationId: 3
2014-10-24 07:37:09,669 INFO  resourcemanager.ClientRMService 
(ClientRMService.java:submitApplication(572)) - Application with id 3 submitted 
by user hrt_qa
2014-10-24 07:37:09,669 INFO  resourcemanager.RMAuditLogger 
(RMAuditLogger.java:logSuccess(147)) - USER=hrt_qa  IP=172.31.6.149 
OPERATION=Submit Application Request    TARGET=ClientRMService  RESULT=SUCCESS  
APPID=application_1414136032036_0003
2014-10-24 07:37:09,820 INFO  delegation.AbstractDelegationTokenSecretManager 
(AbstractDelegationTokenSecretManager.java:renewToken(452)) - Token renewal for 
identifier: owner=hrt_qa, renewer=yarn, 
realUser=oozie/ip-172-31-6-149.ec2.inter...@example.com, 
issueDate=1414136228347, maxDate=1414741028347, sequenceNumber=4, 
masterKeyId=2; total currentTokens 4
2014-10-24 07:37:09,820 WARN  security.DelegationTokenRenewer 
(DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to 
add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: RM_DELEGATION_TOKEN, Service: 
IP:8032,IP:8032, Ident: (owner=hrt_qa, renewer=yarn, 
realUser=oozie/ip-172-31-6-149.ec2.inter...@example.com, 
issueDate=1414136228347, maxDate=1414741028347, sequenceNumber=4, masterKeyId=2)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.AccessControlException: yarn is trying to 
renew a token with wrong password
        at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:476)
        at 
org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.renew(RMDelegationTokenIdentifier.java:110)
        at org.apache.hadoop.security.token.Token.renew(Token.java:377)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:474)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
        at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
        ... 6 more
2014-10-24 07:37:09,821 INFO  rmapp.RMAppImpl 
(RMAppImpl.java:rememberTargetTransitionsAndStoreState(962)) - Updating 
application application_1414136032036_0003 with final state: FAILED
2014-10-24 07:37:09,822 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(704)) - 
appli
{code}
{code}

> Yarn jobs via oozie fail with failed to renew token (secure) or digest 
> mismatch (unsecure) errors when RM is being killed
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-2743
>                 URL: https://issues.apache.org/jira/browse/YARN-2743
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Arpit Gupta
>            Priority: Blocker
>
> During our HA testing we have seen yarn jobs run via oozie fail with failed 
> to renew delegation token errors on secure clusters and digest mismatch 
> errors on un secure clusters



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to