[
https://issues.apache.org/jira/browse/HDFS-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788672#comment-13788672
]
Jing Zhao commented on HDFS-5322:
---------------------------------
Looks like the issue happened during the standby->active transitioning process
of the namenode. The namenode had not finished tailing the editlog, which
contained information about generating the corresponding token, thus failed to
find the token in its local cache.
A possible solution is to let the client side retry the same operation in this
scenario. Thus if the namenode could not find the token during the transition,
it can wrap the InvalidToken exception in the RetriableException and ask the
client to retry.
> HDFS delegation token not found in cache errors seen on secure HA clusters
> --------------------------------------------------------------------------
>
> Key: HDFS-5322
> URL: https://issues.apache.org/jira/browse/HDFS-5322
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 2.1.1-beta
> Reporter: Arpit Gupta
> Assignee: Jing Zhao
>
> While running HA tests we have seen issues were we see HDFS delegation token
> not found in cache errors causing jobs running to fail.
> {code}
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> |2013-10-06 20:14:51,193 INFO [main] mapreduce.Job: Task Id :
> attempt_1381090351344_0001_m_000007_0, Status : FAILED
> Error:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 11 for hrt_qa) can't be found in cache
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
