Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1969#issuecomment-52348177
This is an updated JIRA for the same issue
[SPARK-2290](https://issues.apache.org/jira/browse/SPARK-2290). We established
that, for standalone mode, we don't need to ship the driver's spark home to the
executors, which may not use the same spark home. Instead, we should just use
the `Worker`'s current working directory. However, I am not familiar enough
with Mesos to comment on the need of shipping `SPARK_HOME` there.
@iven There are many other places where we export `SPARK_HOME` in addition
to these two. From a quick grep, I found the following:
```
bin/pyspark:export SPARK_HOME="$FWDIR"
bin/run-example:export SPARK_HOME="$FWDIR"
bin/spark-class:export SPARK_HOME="$FWDIR"
bin/spark-submit:export SPARK_HOME="$(cd `dirname $0`/..; pwd)"
sbin/spark-config.sh:export SPARK_HOME=${SPARK_PREFIX}
```
We need to do the same for all of these places in order for your original
intended behavior to take effect. In the longer run, however, we should just
clean up our usages of `SPARK_HOME`, since in many places we don't actually
have any need to export it (or even use the variable `SPARK_HOME`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
