bharath created SPARK-55330:
-------------------------------
Summary: Unable to create PVC through Spark 4.1.0
Key: SPARK-55330
URL: https://issues.apache.org/jira/browse/SPARK-55330
Project: Spark
Issue Type: Bug
Components: Kubernetes, Spark Submit
Affects Versions: 4.1.1
Environment: TEST
Reporter: bharath
Hi Team,
We have hosted spark on kubernetes environment. The existing spark versions are
3.3.2 and 3.5.2 and the job runs fine with the existing access pod
template.yaml and sparkapplication.yaml.
We have started testing spark 4.1.0 and while running the jobs, it is throwing
the below error
26/01/12 09:46:35 ERROR JavaUtils: Failed to create directory
/apps/application/data/blockmgr-2469dd51-bfd2-478d-9b3c-d5593bf21c26
java.nio.file.AccessDeniedException:
/apps/application/data/blockmgr-2469dd51-bfd2-478d-9b3c-d5593bf21c26
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
at java.base/java.nio.file.Files.createDirectory(Files.java:700)
at
java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
We have created namespaces in TEST region. for 1 each spark 3.3.2, spark 3.5.2
and spark 4.1.0. Issue persists with 4.1.0.
We are using the below conf file in 3.5.2 and 4.1.0 versions.
*spark-defaults.conf*
spark.kubernetes.driver.ownPersistentVolumeClaim
true
spark.kubernetes.driver.reusePersistentVolumeClaim
true
spark.kubernetes.driver.waitToReusePersistentVolumeClaim
true
spark.shuffle.sort.io.plugin.class
org.apache.spark.shuffle.KubernetesLocalDiskShuffleDataIO
spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName
OnDemand
spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass
spark-sc
spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit
5Gi
spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path
/apps/application/data
spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly
false
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName
OnDemand
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass
spark-sc
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit
50Gi
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path
/apps/application/data
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly
false
*pod template file:*
apiVersion: v1
kind: Pod
metadata:
spec:
securityContext:
fsGroup: 4222
fsGroupChangePolicy: OnRootMismatch
containers:
- name: spark-kubernetes-driver
securityContext:
runAsNonRoot: true
capabilities:
drop:
- NET_BIND_SERVICE
seccompProfile:
type: RuntimeDefault
*Dockerfile ( user account and group)*
RUN groupadd -g 4222 spky && \
useradd -u 92461 sparknix -d /home/sparknix -g 4222 -G spky -G root
--no-log-init
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
