[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148042#comment-14148042
]
Jitendra Nath Pandey commented on HDFS-6904:
--------------------------------------------
At a higher level, the issue is as follows:
It has been an assumption that a client gets a delegation token and sets the
service and port at the client side. However, with a non java client using rest
APIs, this assumption breaks down, because the delegation token is a base64
encoded, serialized java object. Therefore, client is unable to set the
appropriate service in the token.
One possible solution:
Provide another rest API in which the client can send a delegation token and
a desired service, port and kind etc, which the server can set inside the token
and return the updated token.
We have scenarios where a smart java client clones the token and sets
different service and ports. This approach will allow a thin client as well to
use tokens in a similar way.
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---------------------------------------------------------------------------------------
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Reporter: Varun Vasudev
> Assignee: Haohui Mai
> Priority: Critical
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token 2222 for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24 more
> 2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher
> (AsyncDispatcher.java:dispatch(164)) - Dispatching the event
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType:
> APP_REJECTED
> {noformat}
> I suspect the issue is that the Namenode generates a delegation token of kind
> WebHDFS but doesn't change the service port. When YARN tries to renew the
> delegation token, it ends up trying to contact WebHDFS on the RPC port.
> From NamenodeWebHdfsMethods.java
> {noformat}
> case GETDELEGATIONTOKEN:
> {
> if (delegation.getValue() != null) {
> throw new IllegalArgumentException(delegation.getName()
> + " parameter is not null.");
> }
> final Token<? extends TokenIdentifier> token = generateDelegationToken(
> namenode, ugi, renewer.getValue());
> final String js = JsonUtil.toJsonString(token);
> return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
> }
> {noformat}
> which in turn calls
> {noformat}
> private Token<? extends TokenIdentifier> generateDelegationToken(
> final NameNode namenode, final UserGroupInformation ugi,
> final String renewer) throws IOException {
> final Credentials c = DelegationTokenSecretManager.createCredentials(
> namenode, ugi, renewer != null? renewer: ugi.getShortUserName());
> final Token<? extends TokenIdentifier> t =
> c.getAllTokens().iterator().next();
> Text kind = request.getScheme().equals("http") ?
> WebHdfsFileSystem.TOKEN_KIND
> : SWebHdfsFileSystem.TOKEN_KIND;
> t.setKind(kind);
> return t;
> }
> {noformat}
> The command we used to get the delegation token is -
> {noformat}
> curl -i -k -s --negotiate -u :
> 'http://NameNodeHost:50070/webhdfs/v1?op=GETDELEGATIONTOKEN&renewer=yarn'
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
