Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19840
I'm trying to understand what is
https://github.com/apache/spark/blob/master/python/pyspark/context.py#L191
really achieving. It seems pretty broken to me and feels like the whole
`pythonExec` tracking in the various places should be removed.
It causes this problem because it forces the executor to use the driver's
python even if it's been set to a different path by the user.
It uses `python` instead of `sys.executable` as the default value.
And it ignores the `spark.pyspark.python` config value if it's set.
Instead, shouldn't the logic at
https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L304
be used in `PythonRunner` (except for the driver python config) to find out
the executor's python to use?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
