[
https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847926#comment-16847926
]
Marcelo Vanzin commented on SPARK-24149:
----------------------------------------
bq. If they are unrelated the user explicitly provides them.
So you basically have the opposite view of Marco's. He thinks that it's better
for users to get tokens automatically if the NN is in the configuration. You
could consider that if the user provides a configuration that references some
NN, then that's explicit enough, and Spark should get tokens for it.
I don't really care either way.
> Automatic namespaces discovery in HDFS federation
> -------------------------------------------------
>
> Key: SPARK-24149
> URL: https://issues.apache.org/jira/browse/SPARK-24149
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.4.0
> Reporter: Marco Gaido
> Assignee: Marco Gaido
> Priority: Minor
> Fix For: 2.4.0
>
>
> Hadoop 3 introduced HDFS federation.
> Spark fails to write on different namespaces when Hadoop federation is turned
> on and the cluster is secure. This happens because Spark looks for the
> delegation token only for the defaultFS configured and not for all the
> available namespaces. A workaround is the usage of the property
> {{spark.yarn.access.hadoopFileSystems}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
