[
https://issues.apache.org/jira/browse/YARN-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev resolved YARN-2426.
---------------------------------
Resolution: Fixed
Fix Version/s: 2.6.0
Fixed with HDFS-6904 exposing an API to allow clients to set the service.
> ResourceManger is not able renew WebHDFS token when application submitted by
> Yarn WebService
> --------------------------------------------------------------------------------------------
>
> Key: YARN-2426
> URL: https://issues.apache.org/jira/browse/YARN-2426
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, resourcemanager, webapp
> Affects Versions: 2.6.0
> Environment: Hadoop Keberos (Secure) cluster with
> LinuxContainerExcutor is enabled
> With SPNEGO on for Yarn new RM web services for application submission
> So during application submission xml/json structure was pass webhdfs token
> Reporter: Karam Singh
> Assignee: Varun Vasudev
> Fix For: 2.6.0
>
>
> Encountered this issue during using new YARN's RM WS for application
> submission, on single node cluster while submitting Distributed Shell
> application using RM WS(webservice).
> For this we need pass custom script and AppMaster jar along with webhdfs
> token.
> Application was failing with ResouceManager was failing to renew token for
> user (appOwner). So RM was Rejecting application with following exception
> trace in RM log:
> {code}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: <NNHOST>:<FSPORT>, Ident: (WEBHDFS delegation token 2222 for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24 more
> 2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher
> (AsyncDispatcher.java:dispatch(164)) - Dispatching the event
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType:
> APP_REJECTED
> {code}
> From exception trace it is clear that RM is try contact to Namenode on FSPort
> instead of Http port and failing to renew token
> Looks like it is because WebHDFS token Namenodes IP and FSPort in delegation
> token instead of http. Causing RM to contact WebHDFS on FSPort and failing to
> renew token
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
