[ https://issues.apache.org/jira/browse/SPARK-43329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719622#comment-17719622 ]
Qian Sun commented on SPARK-43329: ---------------------------------- [~dongjoon] this ticket is duplicated with SPARK-43342 > driver and executors shared same Kubernetes PVC in Spark 3.4+ > ------------------------------------------------------------- > > Key: SPARK-43329 > URL: https://issues.apache.org/jira/browse/SPARK-43329 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.4.0 > Reporter: comet > Priority: Major > > I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below > error. I would like all the executors and driver to mount the same PVC. Is > this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because > otherwise each of the executors will use an unique and separate PVC. > > Error message is "should contain OnDemand or SPARK_EXECUTOR_ID when requiring > multiple executors" > > below is how I enabled it pvc in spark 3.3 and it works, but does not work in > Spark 3.4 > {code:sh} > spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc > > --conf > spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir > > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc > > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir > > > {code} > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org