Repository: spark Updated Branches: refs/heads/branch-2.0 bde545268 -> d293062a4
[SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH from spark-config.sh ## What changes were proposed in this pull request? During startup of Spark standalone, the script file spark-config.sh appends to the PYTHONPATH and can be sourced many times, causing duplicates in the path. This change adds a env flag that is set when the PYTHONPATH is appended so it will happen only one time. ## How was this patch tested? Manually started standalone master/worker and verified PYTHONPATH has no duplicate entries. Author: Bryan Cutler <[email protected]> Closes #15028 from BryanCutler/fix-duplicate-pythonpath-SPARK-17336. (cherry picked from commit c76baff0cc4775c2191d075cc9a8176e4915fec8) Signed-off-by: Sean Owen <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d293062a Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d293062a Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d293062a Branch: refs/heads/branch-2.0 Commit: d293062a4cd04acf48697b1fd6a70fef97b338da Parents: bde5452 Author: Bryan Cutler <[email protected]> Authored: Sun Sep 11 10:19:39 2016 +0100 Committer: Sean Owen <[email protected]> Committed: Sun Sep 11 10:19:49 2016 +0100 ---------------------------------------------------------------------- sbin/spark-config.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/d293062a/sbin/spark-config.sh ---------------------------------------------------------------------- diff --git a/sbin/spark-config.sh b/sbin/spark-config.sh index a7a44cd..b728448 100755 --- a/sbin/spark-config.sh +++ b/sbin/spark-config.sh @@ -26,5 +26,8 @@ fi export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}/conf"}" # Add the PySpark classes to the PYTHONPATH: -export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}" -export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}" +if [ -z "${PYSPARK_PYTHONPATH_SET}" ]; then + export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}" + export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}" + export PYSPARK_PYTHONPATH_SET=1 +fi --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
