L. C. Hsieh created SPARK-34295:
-----------------------------------
Summary: Allow option similar to
mapreduce.job.hdfs-servers.token-renewal.exclude
Key: SPARK-34295
URL: https://issues.apache.org/jira/browse/SPARK-34295
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 3.2.0
Reporter: L. C. Hsieh
MapReduce jobs can instruct YARN to skip renewal of tokens obtained from
certain hosts by specifying the hosts with configuration
mapreduce.job.hdfs-servers.token-renewal.exclude=<host1>,<host2>,..,<hostN>.
But seems Spark lacks of similar option. So the job submission fails if YARN
fails to renew DelegationToken for any of the remote HDFS cluster. The failure
in DT renewal can happen due to many reason like Remote HDFS does not trust
Kerberos identity of YARN etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
