[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HDFS-6904:
------------------------------------------
Priority: Critical (was: Major)
Target Version/s: 2.6.0
This is important for 2.6, given we are trying to get YARN web-services to be
stable for production usage..
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---------------------------------------------------------------------------------------
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Reporter: Varun Vasudev
> Priority: Critical
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token 2222 for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24 more
> 2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher
> (AsyncDispatcher.java:dispatch(164)) - Dispatching the event
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType:
> APP_REJECTED
> {noformat}
> I suspect the issue is that the Namenode generates a delegation token of kind
> WebHDFS but doesn't change the service port. When YARN tries to renew the
> delegation token, it ends up trying to contact WebHDFS on the RPC port.
> From NamenodeWebHdfsMethods.java
> {noformat}
> case GETDELEGATIONTOKEN:
> {
> if (delegation.getValue() != null) {
> throw new IllegalArgumentException(delegation.getName()
> + " parameter is not null.");
> }
> final Token<? extends TokenIdentifier> token = generateDelegationToken(
> namenode, ugi, renewer.getValue());
> final String js = JsonUtil.toJsonString(token);
> return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
> }
> {noformat}
> which in turn calls
> {noformat}
> private Token<? extends TokenIdentifier> generateDelegationToken(
> final NameNode namenode, final UserGroupInformation ugi,
> final String renewer) throws IOException {
> final Credentials c = DelegationTokenSecretManager.createCredentials(
> namenode, ugi, renewer != null? renewer: ugi.getShortUserName());
> final Token<? extends TokenIdentifier> t =
> c.getAllTokens().iterator().next();
> Text kind = request.getScheme().equals("http") ?
> WebHdfsFileSystem.TOKEN_KIND
> : SWebHdfsFileSystem.TOKEN_KIND;
> t.setKind(kind);
> return t;
> }
> {noformat}
> The command we used to get the delegation token is -
> {noformat}
> curl -i -k -s --negotiate -u :
> 'http://NameNodeHost:50070/webhdfs/v1?op=GETDELEGATIONTOKEN&renewer=yarn'
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
