[
https://issues.apache.org/jira/browse/MAPREDUCE-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969458#comment-15969458
]
Michael Gummelt commented on MAPREDUCE-6876:
--------------------------------------------
bq. The job submitting code does not know where the input lives nor how to grab
tokens for it – that's the responsibility of the input format.
That's fine, but it should be factored out into a separate method that the job
submission code can then delegate to. {{listStatus}} does not require
delegation tokens, so it shouldn't fetch delegation tokens.
> FileInputFormat.listStatus should not fetch delegation tokens
> -------------------------------------------------------------
>
> Key: MAPREDUCE-6876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6876
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Michael Gummelt
>
> {{FileInputFormat.listStatus}} fetches delegation tokens:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L213
> AFAICT, this is unnecessary. {{listStatus}} doesn't delegate those tokens to
> another process. This is causing issues described in the attached Spark
> Kerberos ticket, because {{TokenCache.obtainTokensForNameNodes}}, which is
> used to fetch the delegation tokens, assumes that certain MapReduce
> configuration variables are set, which isn't true in the Spark calling code.
> This is a separate problem, but nonetheless it wouldn't have arisen if
> {{listStatus}} weren't fetching delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
