This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 87b312a9c92 [SPARK-39965][K8S] Skip PVC cleanup when driver doesn't
own PVCs
87b312a9c92 is described below
commit 87b312a9c9273535e22168c3da73834c22e1fbbb
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Aug 8 09:58:14 2022 -0700
[SPARK-39965][K8S] Skip PVC cleanup when driver doesn't own PVCs
### What changes were proposed in this pull request?
This PR aims to skip PVC cleanup logic when
`spark.kubernetes.driver.ownPersistentVolumeClaim=false`.
### Why are the changes needed?
To simplify Spark termination log by removing unnecessary log containing
Exception message when Spark jobs have no PVC permission and at the same time
`spark.kubernetes.driver.ownPersistentVolumeClaim` is `false`.
### Does this PR introduce _any_ user-facing change?
Only in the termination logs of Spark jobs that has no PVC permission.
### How was this patch tested?
Manually.
Closes #37433 from dongjoon-hyun/SPARK-39965.
Lead-authored-by: Dongjoon Hyun <[email protected]>
Co-authored-by: pralabhkumar <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../cluster/k8s/KubernetesClusterSchedulerBackend.scala | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index 43c6597362e..985b8b7bef0 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -141,11 +141,13 @@ private[spark] class KubernetesClusterSchedulerBackend(
}
}
- Utils.tryLogNonFatalError {
- kubernetesClient
- .persistentVolumeClaims()
- .withLabel(SPARK_APP_ID_LABEL, applicationId())
- .delete()
+ if (conf.get(KUBERNETES_DRIVER_OWN_PVC)) {
+ Utils.tryLogNonFatalError {
+ kubernetesClient
+ .persistentVolumeClaims()
+ .withLabel(SPARK_APP_ID_LABEL, applicationId())
+ .delete()
+ }
}
if (shouldDeleteExecutors) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
