GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/1734
[SPARK-2454] Do not ship spark home to Workers
When standalone Workers launch executors, they inherit the Spark home set
by the driver. This means if the worker machines do not share the same
directory structure as the driver node, the Workers will attempt to run scripts
(e.g. bin/compute-classpath.sh) that do not exist locally and fail. This is a
common scenario if the driver is launched from outside of the cluster.
The solution is to simply not pass the driver's Spark home to the Workers.
This PR further avoids overloading the usages of `spark.home`, which should now
only be used for setting the executor working directory in Mesos.
This is based on top of #1392 and originally reported by @YanTangZhai.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark spark-home-reprise
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1734.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1734
----
commit 09272b73ea6faab00c30bd3cde161d25e74648b5
Author: Andrew Or <[email protected]>
Date: 2014-08-02T04:14:53Z
Always use Worker's working directory as spark home
commit 188fc5d419870627919cb25179babe12e46cf297
Author: Andrew Or <[email protected]>
Date: 2014-08-02T05:20:08Z
Avoid using spark.home where possible
commit 1c2532c9444ae228f7273517d1ad69529682e229
Author: Andrew Or <[email protected]>
Date: 2014-08-02T05:20:36Z
Merge branch 'master' of github.com:apache/spark into spark-home-reprise
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
