[
https://issues.apache.org/jira/browse/MAPREDUCE-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700263#comment-13700263
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-5375:
----------------------------------------------------
Debugged this with [~venkatnrangan] and [~sseth]. There are a few issues here.
One is the HDFS delegation token renewal issue. Oozie does not handle the auth
to locals mapping configuration correctly. Furthermore, there can be multiple
version of this auth to local configuration in different clusters that Oozie
has (specified by the oozie configuration property
oozie.service.HadoopAccessorService.hadoop.configurations). The entries are
keyed by the host:port of the jt/nn host:port but there is a default catch-all
configuration if specific hadoop configuration is not specified. The fix on
oozie side is to use the oozie catch-all configuration (specified by *= in the
oozie hadoop config mapping entry) to have the right auth to local mapping and
use it in all the clusters that oozie is launching jobs.
There is MR side delegation issue to. Irrespective of the renewer principal
Oozie sets, the JobTracker changes the token to a local name but then expects
it to be a full principal when renewwing. So, obviously Oozie cannot satisfy
this requirement and the MR delegation issue has to be fixed in Jobtracker and
the HDFS delegation issue will be addressed in Oozie.
> Delegation Token renewal exception in jobtracker logs
> -----------------------------------------------------
>
> Key: MAPREDUCE-5375
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5375
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 1.2.0
> Reporter: Venkat Ranganathan
> Assignee: Venkat Ranganathan
> Priority: Critical
>
> Filing on behalf of [~venkatnrangan] who found this originally and provided a
> patch.
> Saw this in the JT logs while oozie tests were running with Hadoop.
> When Oozie java action is executed, the following shows up in the job tracker
> log.
> {code}
> ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
> Exception renewing tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65
> 64 26 6f 6f 7a 69 65 2f 63 6f 6e 64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e
> 6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a 01 3e a6 87 5e 5b 8a 01 3e ca 93
> e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: ip:50300. Not
> rescheduled
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.security.AccessControlException: Client jt/[email protected]
> tries to renew a token with renewer specified as mapred
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267)
> at
> org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)
> at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown
> Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown
> Source)
> at
> org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578)
> at org.apache.hadoop.security.token.Token.renew(Token.java:309)
> at
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221)
> at
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> {code}
> Setting the renewer to Kerberos Local name does not help because
> AbstractDelegationTokenIdentifier sets the renewer to Kerberos shortname but
> JobTracker.renewDelegationToken uses the fullName. This essentially causes
> the renewal to fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
