Yesha Vora created YARN-5098:
--------------------------------
Summary: Yarn Application log Aggreagation fails due to NM can not
get correct HDFS delegation token
Key: YARN-5098
URL: https://issues.apache.org/jira/browse/YARN-5098
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: Yesha Vora
Scenario:
* set dfs.namenode.delegation.token.max-lifetime=43200000 and
dfs.namenode.delegation.token.renew-interval=28800000
* Start long running applications
* Let these applications run for ~3 days
* After 3 days, kill Kill the applications
* try to get application logs for above long running apps.
However, Yarn application logs for long running application could not be
gathered because Nodemanager failed to talk to HDFS with below error.
{code}
2016-05-16 18:18:28,533 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:finishLogAggregation(555)) - Application just
finished : application_1463170334122_0002
2016-05-16 18:18:28,545 WARN ipc.Client (Client.java:run(705)) - Exception
encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 171 for hrt_qa) can't be found in cache
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:583)
at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:398)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:752)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:748)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:747)
at org.apache.hadoop.ipc.Client$Connection.access$3100(Client.java:398)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1597)
at org.apache.hadoop.ipc.Client.call(Client.java:1439)
at org.apache.hadoop.ipc.Client.call(Client.java:1386)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:240)
at com.sun.proxy.$Proxy83.getServerDefaults(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getServerDefaults(ClientNamenodeProtocolTranslatorPB.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy84.getServerDefaults(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.getServerDefaults(DFSClient.java:1018)
at org.apache.hadoop.fs.Hdfs.getServerDefaults(Hdfs.java:156)
at
org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:550)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:687)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
