Hello,
I am trying to run spark jobs using Spark Kubernetes Operator.
But when I try to bundle a conda python environment using the following
resource description the python interpreter is only unpack to the driver and
not to the executors.
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: ...
spec:
type: Python
pythonVersion: "3"
mode: cluster
mainApplicationFile: local:///path/script.py
..
sparkConf:
"spark.archives": "local:///path/conda-env.tar.gz#environment"
"spark.pyspark.python": "./environment/bin/python"
"spark.pyspark.driver.python": "./environment/bin/python"
The driver is unpacking the archive and the python scripts gets executed.
On executors there is no log message indicating that the archive gets unpacked.
Executors then fail as they cant find the python executable at the given
location "./environment/bin/python".
Any hint?
Best,
Meikel
