[
https://issues.apache.org/jira/browse/YUNIKORN-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103024#comment-18103024
]
Dale Richardson commented on YUNIKORN-3128:
-------------------------------------------
We reproduced this deterministically (kind + KWOK, 3,000-pod burst at ~2,000
pods/s, apiserver killed while binds are in flight) on both 1.9.0 and current
master, and traced the full root cause — filed as YUNIKORN-3355 with the
mechanism and file/line references. A few results that may help shape the fix
here:
*1. One correction to the working theory in this thread.* "The core still
thinks the allocation is assigned and will never retry it" was accurate as a
review of PR #995's local-only rollback, but it does not describe master (or
1.7.0): {{releaseAllocation()}} fires on the Failed transition and the core
does release the allocation - in our repro the application drains and completes
normally. The reason the pods are stuck *permanently* is on the shim side: the
failed bind leaves a stale node assignment in the scheduler cache
({{{}forgetPod{}}} re-inserts the assumed pod copy with {{Spec.NodeName}} still
set), and when a later informer event re-creates the app/task, {{updatePod}}
copies that stale assignment onto the pod, so the task registers with the core
as a placed "existing allocation" ({{{}NodeID{}}} set) instead of a pending ask
- never scheduled, never bound, counted against queue quota. Every event that
should rescue the pod converts it into a phantom instead. Details and repro in
YUNIKORN-3355.
*2. Consequence for this ticket: retries and the cache cleanup need to ship
together.* Retries shrink the trigger window (a ~10–15s budget with jitter
fully absorbed our 10s api server restart we used to reproduce the issue), but
any failure longer than the budget still exhausts retries and lands in the
poisoned-cache path - so only adding the retry won't fix the underlying issue.
The YUNIKORN-3355 fixes ({{{}forgetPod{}}} actually reverting the assignment;
{{updatePod}} not stamping stale assignments onto unassigned pods) are the
other half.
*3. Scope note.* Of the four retry points in the 2026-07-03 analysis,
{{AssumePod}} / {{GetPodVolumeClaims}} are already covered since YUNIKORN-3123
(#985). The remaining gaps are {{bindPodVolumes}} and {{Bind}} - both run in
the per-task goroutine, so retrying there is concurrency-safe. (The existing
AssumePod loop runs on the RM callback dispatch thread, where each retrying pod
stalls the whole core→shim pipeline for up to ~30s — worth revisiting while in
this code.)
*4. Error classes.* client-go does not retry non-GET 500s/429s (as Paul noted
on YUNIKORN-2804); we confirmed empirically that a connection-refused Binding
POST fails instantly. The shim retry should cover connection errors, timeouts,
429 (honouring Retry-After) and 500 (etcd apply-lag surfaces as 500), and fail
fast on 404/409/validation.
*5. Terminal behaviour.* On retry exhaustion, resetting the task to pending
with backoff (kube-scheduler prior art: failed bind → ForgetPod + backoff
requeue) is much stronger than terminal Failed — today a bind-failed pod can
only be rescued by an external informer event, which is exactly the path item 1
poisons.
> Yunikorn ignores pending pods after apiserver errors
> ----------------------------------------------------
>
> Key: YUNIKORN-3128
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3128
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - scheduler
> Affects Versions: 1.7.0
> Environment: EKS 1.31
> Reporter: Ruiwen Zhao
> Assignee: Aditya Maheshwari
> Priority: Major
> Labels: pull-request-available
>
> We are running some load testing with Yunikorn, where pods are created at
> 200/s and we monitor if Yunikorn can schedule them at the same rate.
>
> One issue we saw is that Yunikorn ends up ignoring a bunch of (~2000) pods at
> the end of the load test, and completes the application. As shown below,
> there are many pods still Pending, but Yunikorn completes the application
> they belong to, and therefore those pods are stuck. All the pods has
> "schedulerName: yunikorn".
>
> {code:java}
> ❯ kc get pods -n spark8s-kube-burner-yunikorn | grep Pending | head
> kube-burner-0-0-82077 0/1 Pending 0 16m
> kube-burner-0-0-82105 0/1 Pending 0 16m
> kube-burner-0-0-82129 0/1 Pending 0 16m
> kube-burner-0-0-82132 0/1 Pending 0 16m
> kube-burner-0-0-82140 0/1 Pending 0 16m
> kube-burner-0-0-82141 0/1 Pending 0 16m
> 2025-09-29T18:28:18.866Z INFO core.scheduler.fsm
> objects/application_state.go:147 Application state transition {"appID":
> "yunikorn-spark8s-kube-burner-yunikorn-0", "source": "Completing",
> "destination": "Completed", "event": "completeApplication"} {code}
> When looking at one of the Pending pods (kube-burner-0-0-82077), we can
> Yunikorn was trying to schedule it, but failed to do so because of the etcd
> errors. Yunikorn retried once, failed again, and then Yunikorn submitted the
> task again, but no log after that:
> {code:java}
> 2025-09-30T21:18:45.248Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "New",
> "destination": "Pending", "event": "InitTask"}
> 2025-09-30T21:18:45.260Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "Pending",
> "destination": "Scheduling", "event": "SubmitTask"}
> 2025-09-30T21:18:59.464Z ERROR shim.client client/kubeclient.go:127 failed to
> bind pod {"namespace": "spark8s-kube-burner-yunikorn", "podName":
> "kube-burner-0-0-82077", "error": "Operation cannot be fulfilled on
> pods/binding \"kube-burner-0-0-82077\": etcdserver: request timed out"}
> 2025-09-30T21:18:59.465Z ERROR shim.cache.task cache/task.go:464 task failed
> {"appID": "yunikorn-spark8s-kube-burner-yunikorn-0", "taskID":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "reason": "bind pod to node failed,
> name: spark8s-kube-burner-yunikorn/kube-burner-0-0-82077, Operation cannot be
> fulfilled on pods/binding \"kube-burner-0-0-82077\": etcdserver: request
> timed out"}
> 2025-09-30T21:18:59.465Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "Allocated",
> "destination": "Failed", "event": "TaskFail"}
> 2025-09-30T21:18:59.464Z ERROR shim.cache.task cache/task.go:388 bind pod to
> node failed {"taskID": "731dc815-9ee0-4767-a5a9-939219b94f6e", "error":
> "Operation cannot be fulfilled on pods/binding \"kube-burner-0-0-82077\":
> etcdserver: request timed out"}
> 2025-09-30T21:19:25.464Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "New",
> "destination": "Pending", "event": "InitTask"}
> 2025-09-30T21:19:25.464Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "Pending",
> "destination": "Scheduling", "event": "SubmitTask"} {code}
> The failure seems to be caused by etcd timeout, which makes sense, but IMO
> the expected behavior is that Yunikorn keeps trying to schedule the pods with
> backoff.
>
> Yunkorn version: 1.7.0
> Env: EKS 1.31
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
