[
https://issues.apache.org/jira/browse/SPARK-59008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkata krishnan Sowrirajan updated SPARK-59008:
------------------------------------------------
Description:
ExecutorPodsLifecycleManager.removeExecutorFromK8s calls "podToDelete.get()"
twice:
{code:java}
if (podToDelete.get() != null &&
podToDelete.get.getMetadata.getDeletionTimestamp == null) {
{code}
podToDelete is a lazy PodResource handle, so each get() issues a separate
API-server call. If the pod is deleted between the two calls, the second
returns null and dereferencing getMetadata throws:
{code:java}
java.lang.NullPointerException: Cannot invoke
"io.fabric8.kubernetes.api.model.Pod.getMetadata()"
because the return value of
"io.fabric8.kubernetes.client.dsl.PodResource.get()" is null
at
ExecutorPodsLifecycleManager.$anonfun$removeExecutorFromK8s$1(ExecutorPodsLifecycleManager.scala:222)
at
ExecutorPodsLifecycleManager.onFinalNonDeletedState(ExecutorPodsLifecycleManager.scala:194)
at
ExecutorPodsLifecycleManager.onNewSnapshots(ExecutorPodsLifecycleManager.scala:98)
{code}
Observed on a driver managing ~1000 executors during heavy pod churn. The
exception aborts the K8s-side deletion for that executor (Spark-side accounting
in removeExecutorFromSpark has already completed), leaving the pod to be reaped
on a later resync and logging a spurious uncaught-exception error.
Fix: fetch the pod once and test that single value.
was:
ExecutorPodsLifecycleManager.removeExecutorFromK8s calls "podToDelete.get()"
twice:
{code:java}
if (podToDelete.get() != null &&
podToDelete.get.getMetadata.getDeletionTimestamp == null) {
{code}
podToDelete is a lazy PodResource handle, so each get() issues a separate
API-server call. If the pod is deleted between the two calls, the second
returns null and
dereferencing getMetadata throws:
{code:java}
java.lang.NullPointerException: Cannot invoke
"io.fabric8.kubernetes.api.model.Pod.getMetadata()"
because the return value of
"io.fabric8.kubernetes.client.dsl.PodResource.get()" is null
at
ExecutorPodsLifecycleManager.$anonfun$removeExecutorFromK8s$1(ExecutorPodsLifecycleManager.scala:222)
at
ExecutorPodsLifecycleManager.onFinalNonDeletedState(ExecutorPodsLifecycleManager.scala:194)
at
ExecutorPodsLifecycleManager.onNewSnapshots(ExecutorPodsLifecycleManager.scala:98)
{code}
Observed on a driver managing ~1000 executors during heavy pod churn. The
exception aborts the K8s-side deletion for that executor (Spark-side accounting
in
removeExecutorFromSpark has already completed), leaving the pod to be reaped on
a later resync and logging a spurious uncaught-exception error.
Fix: fetch the pod once and test that single value.
> NullPointerException in ExecutorPodsLifecycleManager when pod is deleted
> between get() calls
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-59008
> URL: https://issues.apache.org/jira/browse/SPARK-59008
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 4.1.1
> Reporter: Venkata krishnan Sowrirajan
> Priority: Major
>
> ExecutorPodsLifecycleManager.removeExecutorFromK8s calls "podToDelete.get()"
> twice:
> {code:java}
> if (podToDelete.get() != null &&
> podToDelete.get.getMetadata.getDeletionTimestamp == null) {
> {code}
> podToDelete is a lazy PodResource handle, so each get() issues a separate
> API-server call. If the pod is deleted between the two calls, the second
> returns null and dereferencing getMetadata throws:
> {code:java}
> java.lang.NullPointerException: Cannot invoke
> "io.fabric8.kubernetes.api.model.Pod.getMetadata()"
> because the return value of
> "io.fabric8.kubernetes.client.dsl.PodResource.get()" is null
> at
> ExecutorPodsLifecycleManager.$anonfun$removeExecutorFromK8s$1(ExecutorPodsLifecycleManager.scala:222)
> at
> ExecutorPodsLifecycleManager.onFinalNonDeletedState(ExecutorPodsLifecycleManager.scala:194)
> at
> ExecutorPodsLifecycleManager.onNewSnapshots(ExecutorPodsLifecycleManager.scala:98)
> {code}
> Observed on a driver managing ~1000 executors during heavy pod churn. The
> exception aborts the K8s-side deletion for that executor (Spark-side
> accounting in removeExecutorFromSpark has already completed), leaving the pod
> to be reaped on a later resync and logging a spurious uncaught-exception
> error.
> Fix: fetch the pod once and test that single value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
