[
https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847888#comment-16847888
]
Marcelo Vanzin commented on SPARK-24149:
----------------------------------------
bq. I think we are duplicating the logic here
Do you have any suggestion for how not to duplicate the logic? Because as far
as I remember there's no HDFS API call for "get tokens for all configured name
nodes in my configuration".
> Automatic namespaces discovery in HDFS federation
> -------------------------------------------------
>
> Key: SPARK-24149
> URL: https://issues.apache.org/jira/browse/SPARK-24149
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.4.0
> Reporter: Marco Gaido
> Assignee: Marco Gaido
> Priority: Minor
> Fix For: 2.4.0
>
>
> Hadoop 3 introduced HDFS federation.
> Spark fails to write on different namespaces when Hadoop federation is turned
> on and the cluster is secure. This happens because Spark looks for the
> delegation token only for the defaultFS configured and not for all the
> available namespaces. A workaround is the usage of the property
> {{spark.yarn.access.hadoopFileSystems}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
