[
https://issues.apache.org/jira/browse/MAPREDUCE-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969524#comment-15969524
]
Michael Gummelt commented on MAPREDUCE-6876:
--------------------------------------------
Yea, I really mean {{getSplits}}.
And that proposal sounds perfect.
> FileInputFormat.listStatus should not fetch delegation tokens
> -------------------------------------------------------------
>
> Key: MAPREDUCE-6876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6876
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Michael Gummelt
>
> {{FileInputFormat.listStatus}} fetches delegation tokens:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L213
> AFAICT, this is unnecessary. {{listStatus}} doesn't delegate those tokens to
> another process. This is causing issues described in the attached Spark
> Kerberos ticket, because {{TokenCache.obtainTokensForNameNodes}}, which is
> used to fetch the delegation tokens, assumes that certain MapReduce
> configuration variables are set, which isn't true in the Spark calling code.
> This is a separate problem, but nonetheless it wouldn't have arisen if
> {{listStatus}} weren't fetching delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
