Dongqing Liu created SPARK-26404:
-------------------------------------
Summary: set spark.pyspark.python or PYSPARK_PYTHON doesn't work
in k8s client-cluster mode.
Key: SPARK-26404
URL: https://issues.apache.org/jira/browse/SPARK-26404
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.4.0
Reporter: Dongqing Liu
Neither
conf.set("spark.executorEnv.PYSPARK_PYTHON", "/opt/pythonenvs/bin/python")
nor
conf.set("spark.pyspark.python", "/opt/pythonenvs/bin/python")
works.
Looks like the executor always picks python from PATH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
