comet created SPARK-43329:
-----------------------------
Summary: driver and executors shared same Kubernetes PVC in Spark
3.4+
Key: SPARK-43329
URL: https://issues.apache.org/jira/browse/SPARK-43329
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.4.0
Reporter: comet
I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below
error. I would like all the executors and driver to mount the same PVC. Is
this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because
otherwise each of the executors will use an unique and separate PVC.
"should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors"
below is how I enabled it pvc in spark 3.3 and it works, but does not work in
Spark 3.4
*spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc
\*
*--conf
spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir
\*
*--conf
spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc
\*
*--conf
spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir*
\
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
