GitHub user mgaido91 opened a pull request:
https://github.com/apache/spark/pull/21216
[SPARK-24149][YARN] Retrieve all federated namespaces tokens
## What changes were proposed in this pull request?
Hadoop 3 introduces HDFS federation. This means that multiple namespaces
are allowed on the same HDFS cluster. In Spark, we need to ask the delegation
token for all the namenodes (for each namespace), otherwise accessing any other
namespace different from the default one (for which we already fetch the
delegation token) fails.
The PR adds the automatic discovery of all the namenodes related to all the
namespaces available according to the configs in hdfs-site.xml.
## How was this patch tested?
manual tests in dockerized env
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mgaido91/spark SPARK-24149
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21216.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21216
----
commit dfdd957c15a43bb601b0ca287b7a84e6c326c4c0
Author: Marco Gaido <marcogaido91@...>
Date: 2018-04-29T08:56:29Z
[SPARK-24149][YARN] Retrieve all federated namespaces tokens
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
