Repository: spark Updated Branches: refs/heads/branch-1.0 9ed17ff34 -> ab912271a
Include the sbin/spark-config.sh in spark-executor This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't, as the PYTHONPATH is not set due to changes in ff5be9a4 https://issues.apache.org/jira/browse/SPARK-1725 Author: Bouke van der Bijl <[email protected]> Closes #651 from bouk/include-spark-config-in-mesos-executor and squashes the following commits: b2f1295 [Bouke van der Bijl] Inline PYTHONPATH in spark-executor eedbbcc [Bouke van der Bijl] Include the sbin/spark-config.sh in spark-executor (cherry picked from commit 2fd2752e572921a9010614eb1c1238c493d34a7c) Signed-off-by: Patrick Wendell <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ab912271 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ab912271 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ab912271 Branch: refs/heads/branch-1.0 Commit: ab912271a2f7ab267102c524755c30a9c1785cd0 Parents: 9ed17ff Author: Bouke van der Bijl <[email protected]> Authored: Thu May 8 20:43:37 2014 -0700 Committer: Patrick Wendell <[email protected]> Committed: Thu May 8 20:43:49 2014 -0700 ---------------------------------------------------------------------- sbin/spark-executor | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/ab912271/sbin/spark-executor ---------------------------------------------------------------------- diff --git a/sbin/spark-executor b/sbin/spark-executor index de5bfab..336549f 100755 --- a/sbin/spark-executor +++ b/sbin/spark-executor @@ -19,5 +19,8 @@ FWDIR="$(cd `dirname $0`/..; pwd)" +export PYTHONPATH=$FWDIR/python:$PYTHONPATH +export PYTHONPATH=$FWDIR/python/lib/py4j-0.8.1-src.zip:$PYTHONPATH + echo "Running spark-executor with framework dir = $FWDIR" exec $FWDIR/bin/spark-class org.apache.spark.executor.MesosExecutorBackend
