Rodrigo created SPARK-61890:
-------------------------------
Summary: Kubernetes executor pods stuck in image startup errors
never time out and block replacement indefinitely
Key: SPARK-61890
URL: https://issues.apache.org/jira/browse/SPARK-61890
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 4.3.0
Reporter: Rodrigo
h2. Description
Spark applications on Kubernetes can wait indefinitely when executor pods are
created successfully but their containers cannot start.
We observed this with executor containers stuck in _ImageInspectError_ because
of corrupted image layers on two nodes. The executor JVMs never started or
registered with the driver. However, Spark continued counting their pods toward
the desired executor total and never requested replacements.
Two applications remained in this state for approximately 24 hours, with zero
registered executors and no task execution, until they were externally
terminated.
h3. Observed behavior
The drivers requested their target executor counts: three executors for one
application and one for the other. Kubernetes created and scheduled all four
pods, but each executor container remained in {_}ImageInspectError{_}:
{code:java}
Failed to inspect image "":
rpc error: code = Unknown desc =
size for layer "<sha>" is unknown, failing getSize()
{code}
Both nodes continued reporting {_}Ready{_}, so Kubernetes did not evict the
pods.
The drivers repeatedly logged:
{code:java}
Initial job has not accepted any resources; check your cluster UI
to ensure that workers are registered and have sufficient resources{code}
No executor registered, and Spark did not delete or replace the stuck pods.
h2. Why existing safeguards do not catch this
In the inspected Kubernetes resource manager code:
- *ExecutorPodsSnapshot* maps pods with phase _Pending_ to _PodPending_
without examining container waiting reasons.
- *ExecutorPodsAllocator* removes an executor from _newlyCreatedExecutors_
once its pod appears in a Kubernetes snapshot. Consequently,
_spark.kubernetes.allocation.executor.timeout_ no longer applies.
- Pending pods count toward the desired executor total, preventing replacement
requests when that total is already satisfied.
- Pending-pod cleanup using _executorIdleTimeout_ applies when the pod count
exceeds the target, rather than imposing a startup deadline.
- Heartbeat timeout monitoring begins after executor registration, so it
cannot detect these executors.
- Executor failure limits do not engage because the pods remain pending rather
than entering {_}PodFailed{_}.
h2. Reproduction
Reproduced using an unavailable executor image:
# Submit a Spark application on Kubernetes with executor work to run.
# Configure an executor image
({_}spark.kubernetes.executor.container.image{_}) that cannot start, such as an
unavailable image, while allowing pod creation and scheduling to succeed.
# Observe that the executor container remains in an image-related waiting
state.
# Wait beyond {_}spark.kubernetes.allocation.executor.timeout{_}.
# Observe that Spark neither deletes nor replaces the stuck pod, and the
application continues waiting without registered executors.
h2. Expected behavior
It would be helpful if Spark had a configurable timeout for executor pods that
are created but never register.
One possible approach would be to report the container's waiting reason and
message after the timeout, remove the pod, and retry up to a configurable
limit. If the issue persists, Spark could eventually fail the application with
a clear error message explaining what went wrong.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]