[
https://issues.apache.org/jira/browse/LIVY-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyorgy Gal updated LIVY-781:
----------------------------
Fix Version/s: 0.10.0
(was: 0.9.0)
This issue has been moved to the 0.10.0 release as part of a bulk update. If
you feel this is moved out inappropriately, feel free to provide justification
and reset the Fix Version to 0.9.0.
> PythonInterpreter doesn't honor spark.pyspark.driver.python
> -----------------------------------------------------------
>
> Key: LIVY-781
> URL: https://issues.apache.org/jira/browse/LIVY-781
> Project: Livy
> Issue Type: Bug
> Components: Interpreter
> Affects Versions: 0.7.0
> Reporter: Kohsuke Kawaguchi
> Priority: Major
> Fix For: 0.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [LIVY-418|https://issues.apache.org/jira/browse/LIVY-418] introduced a
> downstream change from
> [SPARK-13081|https://issues.apache.org/jira/browse/SPARK-13081] where
> {{PythonInterpreter}} honors the {{spark.pyspark.python}} configuration. But
> it fails to honor the {{spark.pyspark.driver.python}} that SPARK-13081 also
> introduces.
> As a result, it's not possible to use different path on the driver and the
> executors with Livy, even though it's possible to do so with Spark.
> My understanding is that
> https://github.com/apache/incubator-livy/commit/64d71aa757a93518427f315e908dcf54373948b2
> should start with:
> {code}
> val pythonExec = conf.getOption("spark.pyspark.driver.python")
> .orElse(conf.getOption("spark.pyspark.python"))
> .orElse(sys.env.get(...
> {code}
> ... just like
> https://github.com/apache/spark/commit/7a9e25c38380e6c62080d62ad38a4830e44fe753
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
