Repository: spark Updated Branches: refs/heads/master 191279ce4 -> 2fd2752e5
Include the sbin/spark-config.sh in spark-executor This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't, as the PYTHONPATH is not set due to changes in ff5be9a4 https://issues.apache.org/jira/browse/SPARK-1725 Author: Bouke van der Bijl <[email protected]> Closes #651 from bouk/include-spark-config-in-mesos-executor and squashes the following commits: b2f1295 [Bouke van der Bijl] Inline PYTHONPATH in spark-executor eedbbcc [Bouke van der Bijl] Include the sbin/spark-config.sh in spark-executor Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2fd2752e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2fd2752e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2fd2752e Branch: refs/heads/master Commit: 2fd2752e572921a9010614eb1c1238c493d34a7c Parents: 191279c Author: Bouke van der Bijl <[email protected]> Authored: Thu May 8 20:43:37 2014 -0700 Committer: Patrick Wendell <[email protected]> Committed: Thu May 8 20:43:37 2014 -0700 ---------------------------------------------------------------------- sbin/spark-executor | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/2fd2752e/sbin/spark-executor ---------------------------------------------------------------------- diff --git a/sbin/spark-executor b/sbin/spark-executor index de5bfab..336549f 100755 --- a/sbin/spark-executor +++ b/sbin/spark-executor @@ -19,5 +19,8 @@ FWDIR="$(cd `dirname $0`/..; pwd)" +export PYTHONPATH=$FWDIR/python:$PYTHONPATH +export PYTHONPATH=$FWDIR/python/lib/py4j-0.8.1-src.zip:$PYTHONPATH + echo "Running spark-executor with framework dir = $FWDIR" exec $FWDIR/bin/spark-class org.apache.spark.executor.MesosExecutorBackend
