Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21216#discussion_r187134476
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -200,7 +200,19 @@ object YarnSparkHadoopUtil {
.map(new Path(_).getFileSystem(hadoopConf))
.getOrElse(FileSystem.get(hadoopConf))
- filesystemsToAccess + stagingFS
+ // add the list of available namenodes for all namespaces in HDFS
federation
+ // if ViewFS is enabled, this is skipped as ViewFS already handles
delegation tokens
+ // for its namespaces
+ val hadoopFilesystems = if (stagingFS.getScheme == "viewfs") {
+ Set.empty
+ } else {
+ Option(hadoopConf.get("dfs.nameservices"))
+ .toSeq.flatMap(_.split(","))
+ .flatMap(ns =>
Option(hadoopConf.get(s"dfs.namenode.rpc-address.$ns")))
--- End diff --
style is `.foo { bar => ... }`
This also will not work for HA, since there's no direct "rpc-address" like
this in that case, and you need to use the namespace URI.
You should probably filter out the staging dir FS in that case, too,
although maybe it's already taken care of (since `filesystemsToAccess` is a
set).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
