Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21216#discussion_r187570029
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -200,7 +200,19 @@ object YarnSparkHadoopUtil {
.map(new Path(_).getFileSystem(hadoopConf))
.getOrElse(FileSystem.get(hadoopConf))
- filesystemsToAccess + stagingFS
+ // add the list of available namenodes for all namespaces in HDFS
federation
+ // if ViewFS is enabled, this is skipped as ViewFS already handles
delegation tokens
+ // for its namespaces
+ val hadoopFilesystems = if (stagingFS.getScheme == "viewfs") {
+ Set.empty
+ } else {
+ Option(hadoopConf.get("dfs.nameservices"))
+ .toSeq.flatMap(_.split(","))
+ .flatMap(ns =>
Option(hadoopConf.get(s"dfs.namenode.rpc-address.$ns")))
--- End diff --
Yes, you are right about HA, thanks. I am working on making it working also
for HA. I will update asap.
> You should probably filter out the staging dir FS in that case, too,
although maybe it's already taken care of (since filesystemsToAccess is a set).
yes, it is already taken care since it is a set. I have also this tested in
the UT I added.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
