[
https://issues.apache.org/jira/browse/YUNIKORN-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3355:
--------------------------------------
Description:
If Pod bind calls fail while allocations are in flight (API server outage,
restart, or sustained errors under load), YuniKorn can permanently strand the
affected pods: they remain {{Pending}} in Kubernetes forever, while the core's
queue books them as {*}allocated{*}. The scheduler logs "No outstanding apps
found" and never retries. Pod metadata updates do not resurrect them; only
deleting/recreating the pods or restarting the scheduler recovers. No
WARN/ERROR is logged for the inconsistency.
This is very likely the root cause of YUNIKORN-3128 ("Yunikorn ignores pending
pods after apiserver errors"): we reproduced the identical end state
deterministically and traced the mechanism through the code.
h2. Reproduction (deterministic when timed right; reproduced on 1.9.0 and
master a82e4f92)
# kind cluster (K8s v1.36.1), 500 KWOK fake nodes, YuniKorn standard mode
(Helm defaults, admission controller disabled).
# Create 3,000 pods with {{schedulerName: yunikorn}} at high rate (~2,000
pods/s, client-go at concurrency 16).
# ~3s in, while Binding POSTs are in flight, {{kill -9}} the kube-apiserver;
kubelet restarts it ~10s later.
# Wait for recovery and observe.
The kill must land while binds are in flight — killing before binding starts
recovers cleanly. Any failure mode producing a burst of failed binds should
trigger it (YUNIKORN-3128 saw it from apiserver errors under load, with no full
outage).
h2. Observed end state (master build, 242 of 3,000 pods affected)
* 242 pods {{Pending}} with empty {{{}spec.nodeName{}}}, permanently (observed
>12 min, no recovery).
* Queue REST API: {{root.default}} shows {{allocatedResource pods: 3000}} —
the queue books all 3,000 {*}including the 242 never bound{*}; the applications
list is empty (app completed and was removed).
* Scheduler log: exactly 242 {{added existing allocation}} lines (1:1 with
stuck pods), each with a {{{}targetNode{}}}, ~30-45s after the outage; later
{{{}No outstanding apps found for a while{}}}.
* New pods (a different app) schedule normally — the damage is silent quota
leakage plus the stranded pods.
h2. Root cause (verified against master a82e4f92)
Four-step interaction between the shim cache's assumed-pod handling and the
core's existing-allocation semantics:
# *Assume stamps NodeName on the cached copy.* {{Context.AssumePod}}
deep-copies the pod, sets {{{}assumedPod.Spec.NodeName = node{}}}, and stores
it in the scheduler cache ({{{}pkg/cache/context.go:869-871{}}}). The cache
records the assignment in {{{}assignedPods{}}}.
# *The stale node assignment survives task teardown — by either of two
code-verified paths.* Our repro logs cannot distinguish which ran (container
log rotation consumed the failure window), but both preserve the assignment:
*# The release callback ran.* {{Context.ForgetPod}} fetches the pod from the
scheduler cache — which returns the *assumed copy* with {{Spec.NodeName}} set —
and passes it to {{SchedulerCache.forgetPod}}
({{pkg/cache/context.go:879-884}}). {{forgetPod}} deletes only the
{{assumedPods}} marker and calls {{updatePod(pod)}} with that copy
({{pkg/cache/external/scheduler_cache.go:494-506}}); inside {{updatePod}} the
copy still satisfies {{IsAssignedPod}}, so it is re-added to the node and
re-recorded in {{assignedPods}}. The "forget" is a no-op for the assignment.
*# The release callback never ran.* Under a mass bind-failure storm the async
event pipeline can drop or indefinitely delay the release delivery (the
drop-on-full behaviour of {{rmproxy.enqueueAndCheckFull}} /
{{dispatcher.asyncDispatch}}), in which case {{ForgetPod}} is never invoked and
the assumed state trivially survives — still marked as an active assume.
# Either way, the invariant that matters is: *no path in bind-failure teardown
removes the node assignment (or clears the stamped {{NodeName}}) from the
scheduler cache.* The subsequent steps are identical for both branches.
## The stale assignment is copied onto the real pod object.* On the next
informer event for the pod (watch reconnect / relist after the outage),
{{updatePod}} sees the incoming pod has {{Spec.NodeName == ""}} while
{{assignedPods}} has an entry, and executes {{pod.Spec.NodeName = nodeName}}
("use existing assignment",
{{{}pkg/cache/external/scheduler_cache.go:~355-358{}}}). Note this also
*mutates the shared informer-cache object* (pods are stored by reference via
{{{}utils.Convert2Pod{}}}), poisoning every other consumer of that object.
## Task re-creation turns the stale assignment into a phantom placed
allocation.* The bind-failure storm failed the tasks and completed/removed the
application, so the informer event re-creates the app and task
({{{}ensureAppAndTaskCreated{}}}); {{Task.updateAllocation}} builds the
allocation with {{NodeID: task.pod.Spec.NodeName}}
({{{}pkg/cache/task.go:319-322{}}}, {{{}pkg/common/si_helper.go:134{}}}) — now
non-empty due to step 3. The core's {{PartitionContext.UpdateAllocation}}
treats any allocation with a NodeID as already placed ("handling existing
allocation" / "added existing allocation",
{{{}pkg/scheduler/partition.go:~1240-1259{}}}) and books it against the queue
without ever scheduling or binding it.
The pod is now invisible to the scheduler (no pending ask), unbound in
Kubernetes, and counted against queue quota. When the re-created app later
completes, the phantom allocations remain in queue accounting with no owning
application.
h2. Impact
* Pods stranded Pending indefinitely after any burst of bind failures; silent
— scheduler appears healthy.
* Queue quota silently leaked by phantom allocations; in quota-tight clusters
this can starve a queue completely.
* Shared informer objects mutated (step 3) — undefined behaviour for all other
informer consumers.
* Amplified by scheduler throughput: the faster the core allocates, the more
binds are in flight during a blip, the larger the phantom set (relevant to the
YUNIKORN-3350 throughput work).
h2. Suggested fixes (revised)
# *Primary:* derive {{NodeID}} for core registration only from the *informer's*
view of {{spec.nodeName}} (ground truth of what is bound), never from
cache-internal assumed state ({{si_helper.go:134}} call sites). This breaks the
chain in both branches above.
# *Backstop for branch (b):* expire assumed pods that receive no bind
confirmation (kube-scheduler prior art, ~30s TTL) so lost release callbacks
cannot leave permanent assumed state.
# *Hygiene for branch (a):* {{forgetPod}} must actually revert the assignment
(clear {{NodeName}} on the copy / restore the informer version, remove the
{{assignedPods}} entry); and {{updatePod}} should stamp an existing assignment
only while the pod is still actively assumed ({{assumedPods}} entry present) —
and must never mutate the informer-owned object.
# *Root cause of branch (b), separate scope:* make the release/allocation event
pipeline bounded-blocking instead of drop-on-full so teardown signals cannot be
silently lost.
# Defence in depth unchanged: bounded transient-error bind retry
(YUNIKORN-2804), and a core-side invariant warning when queue allocations exist
with no owning application.
h2. Environment
kind v0.32 / K8s v1.36.1 single control plane, 500 KWOK fake nodes; reproduced
on YuniKorn 1.9.0 (Helm) and a master build (k8shim a82e4f92). Load generator
and repro script available on request.
was:
If Pod bind calls fail while allocations are in flight (API server outage,
restart, or sustained errors under load), YuniKorn can permanently strand the
affected pods: they remain {{Pending}} in Kubernetes forever, while the core's
queue books them as {*}allocated{*}. The scheduler logs "No outstanding apps
found" and never retries. Pod metadata updates do not resurrect them; only
deleting/recreating the pods or restarting the scheduler recovers. No
WARN/ERROR is logged for the inconsistency.
This is very likely the root cause of YUNIKORN-3128 ("Yunikorn ignores pending
pods after apiserver errors"): we reproduced the identical end state
deterministically and traced the mechanism through the code.
h2. Reproduction (deterministic when timed right; reproduced on 1.9.0 and
master a82e4f92)
# kind cluster (K8s v1.36.1), 500 KWOK fake nodes, YuniKorn standard mode
(Helm defaults, admission controller disabled).
# Create 3,000 pods with {{schedulerName: yunikorn}} at high rate (~2,000
pods/s, client-go at concurrency 16).
# ~3s in, while Binding POSTs are in flight, {{kill -9}} the kube-apiserver;
kubelet restarts it ~10s later.
# Wait for recovery and observe.
The kill must land while binds are in flight — killing before binding starts
recovers cleanly. Any failure mode producing a burst of failed binds should
trigger it (YUNIKORN-3128 saw it from apiserver errors under load, with no full
outage).
h2. Observed end state (master build, 242 of 3,000 pods affected)
* 242 pods {{Pending}} with empty {{{}spec.nodeName{}}}, permanently (observed
>12 min, no recovery).
* Queue REST API: {{root.default}} shows {{allocatedResource pods: 3000}} —
the queue books all 3,000 {*}including the 242 never bound{*}; the applications
list is empty (app completed and was removed).
* Scheduler log: exactly 242 {{added existing allocation}} lines (1:1 with
stuck pods), each with a {{{}targetNode{}}}, ~30-45s after the outage; later
{{{}No outstanding apps found for a while{}}}.
* New pods (a different app) schedule normally — the damage is silent quota
leakage plus the stranded pods.
h2. Root cause (verified against master a82e4f92)
Four-step interaction between the shim cache's assumed-pod handling and the
core's existing-allocation semantics:
# *Assume stamps NodeName on the cached copy.* {{Context.AssumePod}}
deep-copies the pod, sets {{{}assumedPod.Spec.NodeName = node{}}}, and stores
it in the scheduler cache ({{{}pkg/cache/context.go:869-871{}}}). The cache
records the assignment in {{{}assignedPods{}}}.
# *ForgetPod does not undo the assignment.* On release after the failed bind,
{{Context.ForgetPod}} fetches the pod from the cache — the assumed copy with
NodeName set — and passes it to {{SchedulerCache.forgetPod}}
({{{}pkg/cache/context.go:879-884{}}}). {{forgetPod}} calls {{updatePod(pod)}}
with that copy and deletes only the {{assumedPods}} marker
({{{}pkg/cache/external/scheduler_cache.go:494-506{}}}). Inside {{updatePod}}
the copy still satisfies {{{}IsAssignedPod{}}}, so it is re-added to the node
and re-recorded in {{{}assignedPods{}}}. Net effect: after a failed bind the
cache permanently believes the pod is assigned.
# *The stale assignment is copied onto the real pod object.* On the next
informer event for the pod (watch reconnect / relist after the outage),
{{updatePod}} sees the incoming pod has {{Spec.NodeName == ""}} while
{{assignedPods}} has an entry, and executes {{pod.Spec.NodeName = nodeName}}
("use existing assignment",
{{{}pkg/cache/external/scheduler_cache.go:~355-358{}}}). Note this also
*mutates the shared informer-cache object* (pods are stored by reference via
{{{}utils.Convert2Pod{}}}), poisoning every other consumer of that object.
# *Task re-creation turns the stale assignment into a phantom placed
allocation.* The bind-failure storm failed the tasks and completed/removed the
application, so the informer event re-creates the app and task
({{{}ensureAppAndTaskCreated{}}}); {{Task.updateAllocation}} builds the
allocation with {{NodeID: task.pod.Spec.NodeName}}
({{{}pkg/cache/task.go:319-322{}}}, {{{}pkg/common/si_helper.go:134{}}}) — now
non-empty due to step 3. The core's {{PartitionContext.UpdateAllocation}}
treats any allocation with a NodeID as already placed ("handling existing
allocation" / "added existing allocation",
{{{}pkg/scheduler/partition.go:~1240-1259{}}}) and books it against the queue
without ever scheduling or binding it.
The pod is now invisible to the scheduler (no pending ask), unbound in
Kubernetes, and counted against queue quota. When the re-created app later
completes, the phantom allocations remain in queue accounting with no owning
application.
h2. Impact
* Pods stranded Pending indefinitely after any burst of bind failures; silent
— scheduler appears healthy.
* Queue quota silently leaked by phantom allocations; in quota-tight clusters
this can starve a queue completely.
* Shared informer objects mutated (step 3) — undefined behaviour for all other
informer consumers.
* Amplified by scheduler throughput: the faster the core allocates, the more
binds are in flight during a blip, the larger the phantom set (relevant to the
YUNIKORN-3350 throughput work).
h2. Suggested fixes (layered; any of the first three breaks the chain)
# {{forgetPod}} must actually revert the assignment: clear {{Spec.NodeName}}
on (a copy of) the pod before re-inserting, or restore the informer version,
and remove the {{assignedPods}} entry.
# {{updatePod}} must not copy a stale assignment onto an unassigned incoming
pod when that assignment came from an assumed (never confirmed bound) pod — and
must never mutate the incoming informer-owned object (copy-on-write instead).
# The shim should only report {{NodeID}} to the core from the *informer's*
view of {{spec.nodeName}} (ground truth of what is actually bound), never from
cache-internal assumed state.
# Defence in depth: bounded retry of transient bind failures (connection
refused / timeout / 429) before failing the task; an assumed-pod expiry
analogous to kube-scheduler's; a core-side invariant warning when queue
allocated resources exist with no owning application.
h2. Environment
kind v0.32 / K8s v1.36.1 single control plane, 500 KWOK fake nodes; reproduced
on YuniKorn 1.9.0 (Helm) and a master build (k8shim a82e4f92). Load generator
and repro script available on request.
> Failed bind leaves a stale node assignment in the shim cache
> ------------------------------------------------------------
>
> Key: YUNIKORN-3355
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3355
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler, shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
>
> If Pod bind calls fail while allocations are in flight (API server outage,
> restart, or sustained errors under load), YuniKorn can permanently strand the
> affected pods: they remain {{Pending}} in Kubernetes forever, while the
> core's queue books them as {*}allocated{*}. The scheduler logs "No
> outstanding apps found" and never retries. Pod metadata updates do not
> resurrect them; only deleting/recreating the pods or restarting the scheduler
> recovers. No WARN/ERROR is logged for the inconsistency.
> This is very likely the root cause of YUNIKORN-3128 ("Yunikorn ignores
> pending pods after apiserver errors"): we reproduced the identical end state
> deterministically and traced the mechanism through the code.
> h2. Reproduction (deterministic when timed right; reproduced on 1.9.0 and
> master a82e4f92)
> # kind cluster (K8s v1.36.1), 500 KWOK fake nodes, YuniKorn standard mode
> (Helm defaults, admission controller disabled).
> # Create 3,000 pods with {{schedulerName: yunikorn}} at high rate (~2,000
> pods/s, client-go at concurrency 16).
> # ~3s in, while Binding POSTs are in flight, {{kill -9}} the kube-apiserver;
> kubelet restarts it ~10s later.
> # Wait for recovery and observe.
> The kill must land while binds are in flight — killing before binding starts
> recovers cleanly. Any failure mode producing a burst of failed binds should
> trigger it (YUNIKORN-3128 saw it from apiserver errors under load, with no
> full outage).
> h2. Observed end state (master build, 242 of 3,000 pods affected)
> * 242 pods {{Pending}} with empty {{{}spec.nodeName{}}}, permanently
> (observed >12 min, no recovery).
> * Queue REST API: {{root.default}} shows {{allocatedResource pods: 3000}} —
> the queue books all 3,000 {*}including the 242 never bound{*}; the
> applications list is empty (app completed and was removed).
> * Scheduler log: exactly 242 {{added existing allocation}} lines (1:1 with
> stuck pods), each with a {{{}targetNode{}}}, ~30-45s after the outage; later
> {{{}No outstanding apps found for a while{}}}.
> * New pods (a different app) schedule normally — the damage is silent quota
> leakage plus the stranded pods.
> h2. Root cause (verified against master a82e4f92)
> Four-step interaction between the shim cache's assumed-pod handling and the
> core's existing-allocation semantics:
> # *Assume stamps NodeName on the cached copy.* {{Context.AssumePod}}
> deep-copies the pod, sets {{{}assumedPod.Spec.NodeName = node{}}}, and stores
> it in the scheduler cache ({{{}pkg/cache/context.go:869-871{}}}). The cache
> records the assignment in {{{}assignedPods{}}}.
> # *The stale node assignment survives task teardown — by either of two
> code-verified paths.* Our repro logs cannot distinguish which ran (container
> log rotation consumed the failure window), but both preserve the assignment:
> *# The release callback ran.* {{Context.ForgetPod}} fetches the pod from the
> scheduler cache — which returns the *assumed copy* with {{Spec.NodeName}} set
> — and passes it to {{SchedulerCache.forgetPod}}
> ({{pkg/cache/context.go:879-884}}). {{forgetPod}} deletes only the
> {{assumedPods}} marker and calls {{updatePod(pod)}} with that copy
> ({{pkg/cache/external/scheduler_cache.go:494-506}}); inside {{updatePod}} the
> copy still satisfies {{IsAssignedPod}}, so it is re-added to the node and
> re-recorded in {{assignedPods}}. The "forget" is a no-op for the assignment.
> *# The release callback never ran.* Under a mass bind-failure storm the async
> event pipeline can drop or indefinitely delay the release delivery (the
> drop-on-full behaviour of {{rmproxy.enqueueAndCheckFull}} /
> {{dispatcher.asyncDispatch}}), in which case {{ForgetPod}} is never invoked
> and the assumed state trivially survives — still marked as an active assume.
> # Either way, the invariant that matters is: *no path in bind-failure
> teardown removes the node assignment (or clears the stamped {{NodeName}})
> from the scheduler cache.* The subsequent steps are identical for both
> branches.
> ## The stale assignment is copied onto the real pod object.* On the next
> informer event for the pod (watch reconnect / relist after the outage),
> {{updatePod}} sees the incoming pod has {{Spec.NodeName == ""}} while
> {{assignedPods}} has an entry, and executes {{pod.Spec.NodeName = nodeName}}
> ("use existing assignment",
> {{{}pkg/cache/external/scheduler_cache.go:~355-358{}}}). Note this also
> *mutates the shared informer-cache object* (pods are stored by reference via
> {{{}utils.Convert2Pod{}}}), poisoning every other consumer of that object.
> ## Task re-creation turns the stale assignment into a phantom placed
> allocation.* The bind-failure storm failed the tasks and completed/removed
> the application, so the informer event re-creates the app and task
> ({{{}ensureAppAndTaskCreated{}}}); {{Task.updateAllocation}} builds the
> allocation with {{NodeID: task.pod.Spec.NodeName}}
> ({{{}pkg/cache/task.go:319-322{}}}, {{{}pkg/common/si_helper.go:134{}}}) —
> now non-empty due to step 3. The core's {{PartitionContext.UpdateAllocation}}
> treats any allocation with a NodeID as already placed ("handling existing
> allocation" / "added existing allocation",
> {{{}pkg/scheduler/partition.go:~1240-1259{}}}) and books it against the queue
> without ever scheduling or binding it.
> The pod is now invisible to the scheduler (no pending ask), unbound in
> Kubernetes, and counted against queue quota. When the re-created app later
> completes, the phantom allocations remain in queue accounting with no owning
> application.
> h2. Impact
> * Pods stranded Pending indefinitely after any burst of bind failures;
> silent — scheduler appears healthy.
> * Queue quota silently leaked by phantom allocations; in quota-tight
> clusters this can starve a queue completely.
> * Shared informer objects mutated (step 3) — undefined behaviour for all
> other informer consumers.
> * Amplified by scheduler throughput: the faster the core allocates, the more
> binds are in flight during a blip, the larger the phantom set (relevant to
> the YUNIKORN-3350 throughput work).
> h2. Suggested fixes (revised)
> # *Primary:* derive {{NodeID}} for core registration only from the
> *informer's* view of {{spec.nodeName}} (ground truth of what is bound), never
> from cache-internal assumed state ({{si_helper.go:134}} call sites). This
> breaks the chain in both branches above.
> # *Backstop for branch (b):* expire assumed pods that receive no bind
> confirmation (kube-scheduler prior art, ~30s TTL) so lost release callbacks
> cannot leave permanent assumed state.
> # *Hygiene for branch (a):* {{forgetPod}} must actually revert the assignment
> (clear {{NodeName}} on the copy / restore the informer version, remove the
> {{assignedPods}} entry); and {{updatePod}} should stamp an existing
> assignment only while the pod is still actively assumed ({{assumedPods}}
> entry present) — and must never mutate the informer-owned object.
> # *Root cause of branch (b), separate scope:* make the release/allocation
> event pipeline bounded-blocking instead of drop-on-full so teardown signals
> cannot be silently lost.
> # Defence in depth unchanged: bounded transient-error bind retry
> (YUNIKORN-2804), and a core-side invariant warning when queue allocations
> exist with no owning application.
> h2. Environment
> kind v0.32 / K8s v1.36.1 single control plane, 500 KWOK fake nodes;
> reproduced on YuniKorn 1.9.0 (Helm) and a master build (k8shim a82e4f92).
> Load generator and repro script available on request.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
