venkata91 opened a new pull request, #58291: URL: https://github.com/apache/spark/pull/58291
### What changes were proposed in this pull request? - Fetch the pod once in `ExecutorPodsLifecycleManager.removeExecutorFromK8s` instead of calling `podToDelete.get()` twice. ### Why are the changes needed? - `podToDelete` is a lazy `PodResource` handle, so each `get()` is a separate API server round trip. If the pod is deleted between the two calls, the second returns `null` and dereferencing `getMetadata` throws `NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Pod.getMetadata()" because the return value of "io.fabric8.kubernetes.client.dsl.PodResource.get()" is null`, thrown from `removeExecutorFromK8s` via `onFinalNonDeletedState` / `onNewSnapshots`. - Observed on a driver managing ~1000 executors during heavy pod churn. The exception aborts the K8s-side deletion for that executor, leaving the pod to be reaped on a later resync. - Introduced by SPARK-54197, which added the `deletionTimestamp` check as a second `get()` call rather than reusing the first result. The `&&` short-circuit guards against the first call returning `null`, but not against the pod disappearing between the two. ### Does this PR introduce _any_ user-facing change? - No. ### How was this patch tested? - New unit test in `ExecutorPodsLifecycleManagerSuite` stubbing `get()` to return the pod then `null` on successive calls. Fails on master with the NPE above, passes with this change. Full suite 10/10. ### Was this patch authored or co-authored using generative AI tooling? - No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
