[
https://issues.apache.org/jira/browse/YARN-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935581#comment-16935581
]
Bibin A Chundatt commented on YARN-9627:
----------------------------------------
[[email protected]]
This issue is more like what do we do with renewal submitted if we have lots of
pending apps.
> DelegationTokenRenewer could block transitionToStandy
> -----------------------------------------------------
>
> Key: YARN-9627
> URL: https://issues.apache.org/jira/browse/YARN-9627
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: krishna reddy
> Assignee: Bibin A Chundatt
> Priority: Critical
> Attachments: YARN-9627.001.patch, YARN-9627.002.patch,
> YARN-9627.003.patch
>
>
> Cluster size: 5K
> Running containers: 55K
> *Scenario*: Largenumber of pending applications (around 50K) and performing
> RM switch over
> Below exception :
> {noformat}
> 2019-06-13 17:39:27,594 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: HDFS_DELEGATION_TOKEN, Service: XXXXXXXXX:1616, Ident: (token
> for root: HDFS_DELEGATION_TOKEN owner=root/[email protected], renewer=yarn,
> realUser=, issueDate=1560361265181, maxDate=1560966065181,
> sequenceNumber=104708, masterKeyId=3);exp=1560533965360;
> apps=[application_1560346941775_20702] in 86397766 ms, appId =
> [application_1560346941775_20702]
> 2019-06-13 17:39:27,609 WARN
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Unable to add the application to the delegation token renewer on recovery.
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:522)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleDTRenewerAppRecoverEvent(DelegationTokenRenewer.java:953)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:79)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:912)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2019-06-13 17:58:20,878 ERROR org.apache.zookeeper.ClientCnxn: Time out error
> occurred for the packet 'clientPath:null serverPath:null finished:false
> header:: 27,4 replyHeader:: 27,4295687588,0 request::
> '/rmstore1/ZKRMStateRoot/RMDTSecretManagerRoot/RMDTMasterKeysRoot/DelegationKey_49,F
> response::
> #31ffffff8a16b74ffffffe129768ffffffdbffffffe949ffffff8dffffffd517ffffffcafffffffa,s{4295423577,4295423577,1560342837789,1560342837789,0,0,0,0,17,0,4295423577}
> '.
> 2019-06-13 17:58:20,877 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: HDFS_DELEGATION_TOKEN, Service:
> XXXXXXXXX:1616, Ident: (token for root: HDFS_DELEGATION_TOKEN
> owner=root/[email protected], renewer=yarn, realUser=,
> issueDate=1560366110990, maxDate=1560970910990, sequenceNumber=111891,
> masterKeyId=3);exp=1560534896413; apps=[application_1560346941775_28115]]
> 2019-06-13 17:58:20,924 WARN
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Unable to add the application to the delegation token renewer on recovery.
> java.lang.IllegalStateException: Timer already cancelled.
> at java.util.Timer.sched(Timer.java:397)
> at java.util.Timer.schedule(Timer.java:208)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:612)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:523)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleDTRenewerAppRecoverEvent(DelegationTokenRenewer.java:953)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:79)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:912)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
