anxu created SPARK-17336:
----------------------------
Summary: Repeated calls sbin/spark-config.sh file Causes
${PYTHONPATH} Value duplicate
Key: SPARK-17336
URL: https://issues.apache.org/jira/browse/SPARK-17336
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.0.0
Reporter: anxu
On Spark start up by command: sbin/start-all.sh, the sbin/spark-config.sh
Repeated calls. In sbin/spark-config.sh code.
{code:title=sbin/spark-config.sh|borderStyle=solid}
# Add the PySpark classes to the PYTHONPATH:
export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
{code}
{color:red}PYTHONPATH{color} has duplicate Value.
example:
{code:borderStyle=solid}
axu4iMac:spark-2.0.0-hadoop2.4 axu$ sbin/start-all.sh | grep PYTHONPATH
axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1):
[/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2):
[/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1):
[/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2):
[/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1):
[/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2):
[/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
