[
https://issues.apache.org/jira/browse/YARN-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787752#comment-16787752
]
Tarun Parimi commented on YARN-9224:
------------------------------------
On letting the patch run for a week in my test setup, I see errors in renewing
timeline delegation tokens. It seems the ugi credentials expire and cause the
following exception when RM tries to renew token after 2 days. So this approach
of caching with ugi does not work as now. Cancelling the existing patch.
{code:java}
java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN,
Service: 172.26.74.204:8188, Ident: (owner=ambari-qa, renewer=yarn, realUser=,
issueDate=1551944614055, maxDate=1552549414055, sequenceNumber=1704,
masterKeyId=311)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:495)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$900(DelegationTokenRenewer.java:79)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:919)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:896)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: HTTP status [500], message [Null user]
at
org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:301)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:217)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:415)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:409)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:390)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientRetryOpForOperateDelegationToken.run(TimelineClientImpl.java:702)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:186)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:465)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:412)
at
org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier$Renewer.renew(TimelineDelegationTokenIdentifier.java:95)
at org.apache.hadoop.security.token.Token.renew(Token.java:414)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:617)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:614)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:613)
{code}
> TimelineDelegationTokenIdentifier.Renewer contacts KDC for every renew/cancel
> token operation
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-9224
> URL: https://issues.apache.org/jira/browse/YARN-9224
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.6.0, 2.7.3
> Reporter: Tarun Parimi
> Assignee: Tarun Parimi
> Priority: Major
> Attachments: YARN-9224.001.patch, YARN-9224.002.patch,
> YARN-9224.003.patch
>
>
> In a production cluster, we have observed the active RM principal making
> excessive requests to the KDC server. Being a service principal, this
> shouldn't be the case normally.
> On capturing tcpdump for the connections between RM and KDC, we saw that
> these excessive requests were for the SPNEGO service
> HTTP/ats-host.example.com .
> The requests were also matching in frequency with the below log entry in RM.
> {code:java}
> 2019-01-09T03:41:56.048-0500 INFO
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl: Timeline service
> address: http://ats-host.example.com:8188/ws/v1/timeline/
> {code}
> On looking at the code in TimelineDelegationTokenIdentifier.java, it seems
> this kdc request for SPNEGO is done as we are creating a new timeline client
> instance every time.
> {code:java}
> @SuppressWarnings("unchecked")
> @Override
> public long renew(Token<?> token, Configuration conf) throws IOException,
> InterruptedException {
> TimelineClient client = TimelineClient.createTimelineClient();
> try {
> client.init(conf);
> client.start();
> return client.renewDelegationToken(
> (Token<TimelineDelegationTokenIdentifier>) token);
> } catch (YarnException e) {
> throw new IOException(e);
> } finally {
> client.stop();
> }
> }
> @SuppressWarnings("unchecked")
> @Override
> public void cancel(Token<?> token, Configuration conf) throws IOException,
> InterruptedException {
> TimelineClient client = TimelineClient.createTimelineClient();
> try {
> client.init(conf);
> client.start();
> client.cancelDelegationToken(
> (Token<TimelineDelegationTokenIdentifier>) token);
> } catch (YarnException e) {
> throw new IOException(e);
> } finally {
> client.stop();
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
