Eric Chang created TOREE-310:
--------------------------------
Summary: Allow override of python executable used for pyspark
Key: TOREE-310
URL: https://issues.apache.org/jira/browse/TOREE-310
Project: TOREE
Issue Type: Improvement
Reporter: Eric Chang
We're using virtualenvs when running pyspark, and it would be great to be able
to use a virtualenv as the python executable used by the Spark Driver (i.e.
{{--master yarn-client}}). This value is currently hard-coded to {{python}} in
org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala.
I have a branch on my repo which adds an optional kernel parameter PYTHON_EXEC:
{code}
...
"SPARK_HOME": "/usr/lib/spark",
"PYTHON_EXEC" : "/usr/local/python/virtualenvs/myvenv/bin/python",
...
{code}
Here's the diff of the branch, please let me know if it's ok for me to issue a
PR against the main repo:
https://github.com/ericchang/incubator-toree/compare/ericchang:master...custom-python-exec
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
