Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19840#discussion_r154028953
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -57,9 +57,14 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
require(funcs.length == argOffsets.length, "argOffsets should have the
same length as funcs")
+ private val env = SparkEnv.get
+ private val conf = env.conf
+
// All the Python functions should have the same exec, version and
envvars.
protected val envVars = funcs.head.funcs.head.envVars
- protected val pythonExec = funcs.head.funcs.head.pythonExec
+ protected val pythonExec =
conf.getOption("spark.executorEnv.PYSPARK_DRIVER_PYTHON")
--- End diff --
Shouldn't this be handled in `deploy/PythonRunner`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
