Fei Wang created SPARK-46091:
--------------------------------
Summary: [KUBERNETES] Respect the existing kubernetes container
SPARK_LOCAL_DIRS env
Key: SPARK-46091
URL: https://issues.apache.org/jira/browse/SPARK-46091
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 3.5.0
Reporter: Fei Wang
Respect the user defined SPARK_LOCAL_DIRS container env when setup local dirs.
For example, we use hostPath for spark local dir.
But we do not mount the sub disks directly to the pod, we mount a root path for
spark driver/executor pod.
For example, the root path is `/hadoop`.
And there are sub disks under that, likes `hadoop/1, /hadoop/2, /hadoop/3,
/hadoop4`.
And we want to define the SPARK_LOCAL_DIRS in the driver/executor pod env.
But now, the user specified SPARK_LOCAL_DIRS does not work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
