GitHub user NathanHowell opened a pull request:
https://github.com/apache/spark/pull/257
Minimize the scope of runAsUser to support Hadoop token passing
To support accessing a secure HDFS installation from Mesos executed Spark
jobs, the resources attached to the `SparkContext` should be fetched before
invoking `UserGroupInformation.doAs`. Or at least the credentials should be
fetched before `doAs`. This allows us to set the `HADOOP_TOKEN_FILE_LOCATION`
environment variable on the executor to a token cache file generated for each
job.
`UserGroupInformation.doAs` fails to execute if the environment variable is
set but the file does not exist, and trying to stuff the credentials back in
later via some other mechanism is problematic. This approach is similar to how
credentials are passed for YARN executed jobs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NathanHowell/spark v0.9.1-rc1-gd1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/257.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #257
----
commit 42b9f30ad88d5fc74a4857889c419651306cb10f
Author: Nathan Howell <[email protected]>
Date: 2014-03-27T01:10:49Z
Minimize the scope of runAsUser to support token passing
Need to defer the runAsUser until after files/jars have been fetched.
This is to support the HADOOP_TOKEN_FILE_LOCATION environment variable.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
