HwangDongJun commented on code in PR #58187:
URL: https://github.com/apache/spark/pull/58187#discussion_r4163435261
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala:
##########
@@ -209,6 +293,54 @@ private[spark] class Client(
log"submission ID ${MDC(SUBMISSION_ID, sId)} into Kubernetes")
}
}
+
+ // SPARK-38079: best-effort cleanup for pre-resources (and the driver pod,
if it was created)
+ // that were left without an owner reference because this process was
terminated abruptly
+ // before the owner-reference refresh in run() completed. See the comment
above the shutdown
+ // hook registration in run() for why this window exists and why it must
stop mattering once
+ // that refresh completes.
+ //
+ // package-private (rather than private) so it can be unit-tested directly
-- actually
+ // triggering a JVM shutdown hook from a test is impractical, so tests
instead call this
+ // method directly with injected pre/post-state and a fake
recoveryClientFactory.
+ private[submit] def cleanupOrphanedPreResources(
+ preResources: Seq[HasMetadata],
+ driverPodName: String,
+ preResourcesApplied: Boolean,
+ podCreatedByUs: Boolean): Unit = {
+ if (preResourcesApplied) {
+ try {
+ Utils.tryWithResource(recoveryClientFactory()) { recoveryClient =>
+ try {
+ recoveryClient.resourceList(preResources: _*).delete()
Review Comment:
Done. `deletePodAndPreResources` now deletes each pre-resource independently:
- The pod and every pre-resource each get their own
`kubernetesClient.resource(...).delete()` call, instead of one
`resourceList(...).delete()` for the whole list.
- Each call is wrapped in its own `Utils.tryLogNonFatalError`, so one item
failing (e.g. the `podgroups` 403 case you described) doesn't skip the rest —
it just gets logged and the loop moves on.
This is now the only cleanup path (used at all three catch-block call
sites), so it covers the former L224/L234/L246 cases too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
