Xianyin Xin created YARN-5367:
---------------------------------
Summary: HDFS delegation tokens in ApplicationSubmissionContext
should be added to systemCrednetials
Key: YARN-5367
URL: https://issues.apache.org/jira/browse/YARN-5367
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: Xianyin Xin
Assignee: Xianyin Xin
App log aggregation may failed because of the below flow:
0) suppose the token.max-lifetime is 7 days and renew interval is 1 day;
1) start a long running job, like sparkJDBC, of which the AM acts as a service.
When submitting the job, HDFS token A in ApplicationSubmissionContext will be
added to DelegationTokenRenewer, but not added to systemCredentials;
2) after 1 day, submit a spark query. After received the query, AM will request
containers and start tasks. When start the containers, a new HDFS token B is
used;
3) after 1 day, kill the job, when doing log aggregation, exception occurs
which show token B is not in the HDFS token cache so the connecting to HDFS
fails;
We should add token A to systemCredentials to make sure token A can be
delivered to NMs in time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
