Rodrigo created SPARK-61890:
-------------------------------

             Summary: Kubernetes executor pods stuck in image startup errors 
never time out and block replacement indefinitely
                 Key: SPARK-61890
                 URL: https://issues.apache.org/jira/browse/SPARK-61890
             Project: Spark
          Issue Type: Bug
          Components: Kubernetes
    Affects Versions: 4.3.0
            Reporter: Rodrigo


h2. Description

Spark applications on Kubernetes can wait indefinitely when executor pods are 
created successfully but their containers cannot start.

We observed this with executor containers stuck in _ImageInspectError_ because 
of corrupted image layers on two nodes. The executor JVMs never started or 
registered with the driver. However, Spark continued counting their pods toward 
the desired executor total and never requested replacements.

Two applications remained in this state for approximately 24 hours, with zero 
registered executors and no task execution, until they were externally 
terminated.
h3. Observed behavior

The drivers requested their target executor counts: three executors for one 
application and one for the other. Kubernetes created and scheduled all four 
pods, but each executor container remained in {_}ImageInspectError{_}:

 
{code:java}
Failed to inspect image "":
rpc error: code = Unknown desc =
size for layer "<sha>" is unknown, failing getSize()
{code}
 

Both nodes continued reporting {_}Ready{_}, so Kubernetes did not evict the 
pods.

The drivers repeatedly logged:
{code:java}
Initial job has not accepted any resources; check your cluster UI
to ensure that workers are registered and have sufficient resources{code}
 

No executor registered, and Spark did not delete or replace the stuck pods.
h2. Why existing safeguards do not catch this

In the inspected Kubernetes resource manager code:
 - *ExecutorPodsSnapshot* maps pods with phase _Pending_ to _PodPending_ 
without examining container waiting reasons.
 - *ExecutorPodsAllocator* removes an executor from _newlyCreatedExecutors_ 
once its pod appears in a Kubernetes snapshot. Consequently, 
_spark.kubernetes.allocation.executor.timeout_ no longer applies.
 - Pending pods count toward the desired executor total, preventing replacement 
requests when that total is already satisfied.
 - Pending-pod cleanup using _executorIdleTimeout_ applies when the pod count 
exceeds the target, rather than imposing a startup deadline.
 - Heartbeat timeout monitoring begins after executor registration, so it 
cannot detect these executors.
 - Executor failure limits do not engage because the pods remain pending rather 
than entering {_}PodFailed{_}.

h2. Reproduction

Reproduced using an unavailable executor image:
 # Submit a Spark application on Kubernetes with executor work to run.
 # Configure an executor image 
({_}spark.kubernetes.executor.container.image{_}) that cannot start, such as an 
unavailable image, while allowing pod creation and scheduling to succeed.
 # Observe that the executor container remains in an image-related waiting 
state.
 # Wait beyond {_}spark.kubernetes.allocation.executor.timeout{_}.
 # Observe that Spark neither deletes nor replaces the stuck pod, and the 
application continues waiting without registered executors.

h2. Expected behavior

It would be helpful if Spark had a configurable timeout for executor pods that 
are created but never register.

One possible approach would be to report the container's waiting reason and 
message after the timeout, remove the pod, and retry up to a configurable 
limit. If the issue persists, Spark could eventually fail the application with 
a clear error message explaining what went wrong.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to