LuciferYang commented on code in PR #58187:
URL: https://github.com/apache/spark/pull/58187#discussion_r3849746182
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala:
##########
@@ -136,7 +136,10 @@ private[spark] class Client(
val driverPodName = resolvedDriverPod.getMetadata.getName
// setup resources before pod creation
- val preKubernetesResources =
resolvedDriverSpec.driverPreKubernetesResources
+ // SPARK-38079: the driver's own base config map (mounted as
SPARK_CONF_VOLUME_DRIVER
+ // above) must also be created before the pod itself, to avoid a
"configmap ... not
+ // found" mount race between the driver pod and the config map it depends
on.
+ val preKubernetesResources =
resolvedDriverSpec.driverPreKubernetesResources ++ Seq(configMap)
Review Comment:
With these resources now created before the pod, the first serverSideApply
persists them without an owner reference; the refresh call only adds it once
the pod exists. If spark-submit dies inside that window (SIGKILL, OOM, or a
dropped connection where the catch cleanup's delete also fails), the
keytab/delegation-token secrets and the config maps remain ownerless in the
namespace and are never garbage-collected. On master this state did not exist.
The window is a few seconds, but it now holds credential-bearing secrets.
Two ways out: a shutdown hook in Client — it must build its own
KubernetesClient (once run() throws, Utils.tryWithResource has already closed
the injected one), and it may act only before the owner-reference refresh apply
completes: delete the pre-resources then, plus the pod by name only if this
submission created it (else it can kill an earlier submission's same-named
pod); after that refresh the hook must do nothing, so Ctrl-C in the
waitAppCompletion loop keeps today's detach semantics. Flags volatile,
exceptions swallowed. Or accept the window and state the trade-off in the PR
description. SIGKILL and API calls in flight at shutdown remain uncovered
either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
