[
https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699973#comment-14699973
]
Davies Liu commented on SPARK-9705:
-----------------------------------
As the exception said, PySpark cannot run with different minor versions, you
need to use the same minor version of Python, for example, use both Python 3.4
on driver and workers.
Be default, PySpark use the default `python` found in PATH, if you have
different default python in driver and worker, then you need to specify
PYSPARK_PYTHON to tell pyspark which version of Python you want to use. For
example:
{code}
PYSPARK_PYTHON=the_path_of_python_3.4 bin/spark-submit xxx
{code}
> outdated Python 3 and IPython information
> -----------------------------------------
>
> Key: SPARK-9705
> URL: https://issues.apache.org/jira/browse/SPARK-9705
> Project: Spark
> Issue Type: Documentation
> Components: Documentation, PySpark
> Affects Versions: 1.4.0, 1.4.1, 1.5.0
> Reporter: Piotr Migdał
> Priority: Blocker
> Labels: documentation
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> https://issues.apache.org/jira/browse/SPARK-4897 adds Python 3.4 support to
> 1.4.0 and above, but the official docs (1.4.1, but the same is for 1.4.0)
> says explicitly:
> "Spark 1.4.1 works with Python 2.6 or higher (but not Python 3)."
> Affected:
> https://spark.apache.org/docs/1.4.0/programming-guide.html
> https://spark.apache.org/docs/1.4.1/programming-guide.html
> There are some other Python-related things, which are outdated, e.g. this
> line:
> "For example, to launch the IPython Notebook with PyLab plot support:"
> (At least since IPython 3.0 PyLab/Matplotlib support happens inside a
> notebook; and the line "--pylab inline" is already removed.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
