[
https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851896#comment-16851896
]
Marco Gaido commented on SPARK-24149:
-------------------------------------
That's true, the point is: if you want to access a different cluster, it is
pretty natural to think that you need to add information for accessing
credentials. But it is pretty weird if accessing different paths of the same
cluster, you have to specify several configs.
> Automatic namespaces discovery in HDFS federation
> -------------------------------------------------
>
> Key: SPARK-24149
> URL: https://issues.apache.org/jira/browse/SPARK-24149
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.4.0
> Reporter: Marco Gaido
> Assignee: Marco Gaido
> Priority: Minor
> Fix For: 2.4.0
>
>
> Hadoop 3 introduced HDFS federation.
> Spark fails to write on different namespaces when Hadoop federation is turned
> on and the cluster is secure. This happens because Spark looks for the
> delegation token only for the defaultFS configured and not for all the
> available namespaces. A workaround is the usage of the property
> {{spark.yarn.access.hadoopFileSystems}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
