[
https://issues.apache.org/jira/browse/SPARK-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454745#comment-15454745
]
Sean Owen commented on SPARK-17336:
-----------------------------------
[~axu4apache] I assume you modified the scripts to print PYTHONPATH in order to
show this behavior.
Yeah it looks like the file that sets it is repeatedly sourced so this path is
appended many times.
I don't know of a cleaner way to do this other than to see if the variable
already contains the path and only append if it doesn't. Do you want to try
that in a PR?
> Repeated calls sbin/spark-config.sh file Causes ${PYTHONPATH} Value duplicate
> -----------------------------------------------------------------------------
>
> Key: SPARK-17336
> URL: https://issues.apache.org/jira/browse/SPARK-17336
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.0.0
> Reporter: anxu
>
> On Spark start up by command: sbin/start-all.sh, the sbin/spark-config.sh
> Repeated calls. In sbin/spark-config.sh code.
> {code:title=sbin/spark-config.sh|borderStyle=solid}
> # Add the PySpark classes to the PYTHONPATH:
> export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
> export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
> {code}
> {color:red}PYTHONPATH{color} has duplicate Value.
> example:
> {code:borderStyle=solid}
> axu4iMac:spark-2.0.0-hadoop2.4 axu$ sbin/start-all.sh | grep PYTHONPATH
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1):
> [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2):
> [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1):
> [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2):
> [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1):
> [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2):
> [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
