Venkata krishnan Sowrirajan created SPARK-59008:
---------------------------------------------------
Summary: NullPointerException in ExecutorPodsLifecycleManager when
pod is deleted between get() calls
Key: SPARK-59008
URL: https://issues.apache.org/jira/browse/SPARK-59008
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 4.1.1
Reporter: Venkata krishnan Sowrirajan
ExecutorPodsLifecycleManager.removeExecutorFromK8s calls "podToDelete.get()"
twice:
{code:java}
if (podToDelete.get() != null &&
podToDelete.get.getMetadata.getDeletionTimestamp == null) {
{code}
podToDelete is a lazy PodResource handle, so each get() issues a separate
API-server call. If the pod is deleted between the two calls, the second
returns null and
dereferencing getMetadata throws:
{code:java}
java.lang.NullPointerException: Cannot invoke
"io.fabric8.kubernetes.api.model.Pod.getMetadata()"
because the return value of
"io.fabric8.kubernetes.client.dsl.PodResource.get()" is null
at
ExecutorPodsLifecycleManager.$anonfun$removeExecutorFromK8s$1(ExecutorPodsLifecycleManager.scala:222)
at
ExecutorPodsLifecycleManager.onFinalNonDeletedState(ExecutorPodsLifecycleManager.scala:194)
at
ExecutorPodsLifecycleManager.onNewSnapshots(ExecutorPodsLifecycleManager.scala:98)
{code}
Observed on a driver managing ~1000 executors during heavy pod churn. The
exception aborts the K8s-side deletion for that executor (Spark-side accounting
in
removeExecutorFromSpark has already completed), leaving the pod to be reaped on
a later resync and logging a spurious uncaught-exception error.
Fix: fetch the pod once and test that single value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
