[
https://issues.apache.org/jira/browse/YUNIKORN-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3375:
--------------------------------------
Description:
When a Node object is deleted and re-created while its pods keep running, the
shim and the core end up permanently disagreeing about what is on that node.
The shim has the pods placed; the core believes the node is empty and has given
the queue its quota back. The queue's usage stays understated for as long as
the pods run.
*What works (this issue is not about these)*:
- The orphaned pod tracking works. On node removal the pods are orphaned, and
when the node comes back the adoption path puts them back on the node in the
shim cache correctly.
- There is no node-level double allocation. The shim's own predicates run
against the shim cache, which contains the adopted pods, so a genuinely full
node rejects new placements. We verified this.
*The gap*: adoption only restores the shim's view. Nothing restores the core's.
# On node removal the core removes the allocations from the applications and
decrements the queue's allocated resources ({{removeNodeAllocations}} ->
{{queue.DecAllocatedResource}}), then notifies the shim with {{TerminationType
STOPPED_BY_RM}}. The shim callback correctly only forgets the pod in its cache
and does not fail the task -- the pod is still running, and dispatching the
release would delete it. The task stays Bound.
# On node re-add, the node registration cannot carry allocations
({{existingAllocations}} was removed from the SI in
[YUNIKORN-2780|https://issues.apache.org/jira/browse/YUNIKORN-2780]), and the
adoption path never sends an allocation update either: {{updateYuniKornPod}}
finds the still-Bound task and only calls {{SetTaskPod}}.
# No reconciliation exists. The divergence lasts until each pod terminates, or
until a scheduler restart (restart recovery does re-report).
*Why we believe this is an oversight and not a design decision*: the same
adoption loop DOES send an allocation update to the core for non-YuniKorn pods
({{updateForeignPod}} -> {{CreateAllocationForForeignPod}}, tracking added by
[YUNIKORN-2834|https://issues.apache.org/jira/browse/YUNIKORN-2834]). Only the
YuniKorn-managed branch is silent, because the surviving Bound task suppresses
it.
*Impact*:
- Queue accounting is silently wrong: a queue with max=X can end up running
more than X (the surviving pods plus a full X of new admissions elsewhere).
User quotas, fair-share ordering, preemption inputs, autoscaler calculations,
metrics and the web UI all compute against understated usage.
- If ALL of an application's allocations were on the removed node, it is worse:
the core application is left empty, moves Completing -> Completed (~30s
default), and is removed from the partition entirely. The shim still holds the
application with Bound tasks; the core no longer knows it exists.
- The core also wastes scheduling cycles proposing a node it believes is empty
while the shim rejects every attempt.
*Reachability*: {{kubectl delete node}} on a live node (or automation doing the
equivalent) followed by kubelet self-re-registration recreates the Node object
within seconds. The pod-GC controller quarantines orphaned pods for ~40s before
deleting them, so the pods surviving a brief flap is the normal outcome.
Cluster-autoscaler is not a trigger (it drains first). Rare but ordinary
operations; when it fires it affects every pod on the node.
*Reproduction*: deterministic via the
[YUNIKORN-3377|https://issues.apache.org/jira/browse/YUNIKORN-3377]
fault-injection test ({{TestChaosNodeFlap}}, oracle O5): after a flap, e.g. one
node shows 40 pods placed in the shim and zero allocations in the core, on
every seed.
*Suggested fix*: on adoption of a YuniKorn-managed pod whose task is Bound,
re-report the allocation the way the foreign-pod branch and restart recovery
already do ({{UpdateAllocation}} with the node ID set -> the core's
{{MarkPreviouslyAllocated}} path). Because the core application may already
have been removed (see impact above), the fix must follow the recovery sequence
-- re-register the application first, then report the allocations -- rather
than sending the allocation update alone.
was:
When a Node object is deleted and re-created while its pods keep running, the
shim and the core permanently disagree about the node's allocations, and queue
accounting is silently corrupted.
*Mechanism* (all verified in code):
# Node delete: the shim orphans the node's bound pods
({{{}Context.deleteNodeInternal{}}} -> {{{}SchedulerCache.RemoveNode{}}}); the
core removes the node, removes the allocations from the applications and
*decrements queue allocated resources*
({{{}partition.removeNodeAllocations{}}}), then notifies the shim with
{{{}TerminationType STOPPED_BY_RM{}}}.
# The shim callback intentionally only forgets the pod in the cache and does
not fail the task ({{{}scheduler_callback.go{}}} ~117) – this filter is correct
and must stay: dispatching the release would delete running pods.
# Node re-add: {{registerNodes}} sends a bare {{si.NodeInfo}} (it cannot carry
allocations – {{existingAllocations}} is a reserved/removed SI field), and the
orphan-adoption path re-attaches the pods to the shim cache via
{{updateYuniKornPod}} -> {{SetTaskPod}} on the still-Bound task – {*}nothing
re-reports the allocations to the core{*}.
# No reconciliation exists. The divergence lasts until each pod terminates, or
until a scheduler restart (restart recovery does re-report).
{*}The asymmetry that shows this is an oversight{*}: the same adoption path
already re-reports *foreign* pods to the core ({{{}updateForeignPod{}}} ->
{{CreateAllocationForForeignPod}} -> {{{}UpdateAllocation{}}}). Only the
YuniKorn-managed branch is silent, because the existing Bound task suppresses
task creation.
*Impact* :
- Node-level double allocation does NOT occur: the shim predicate callback
runs {{NodeResourcesFit}} against the shim cache, which contains the adopted
pods, so a genuinely full node rejects new placements (kubelet admission is a
second backstop).
- The real damage is {*}accounting integrity{*}: the queue's allocated
resource was decremented while the pods keep running, and is never
re-incremented. A queue with max=X can end up running more than X (survivors
plus a full X of new admissions); user quotas, fair-share ordering, preemption
inputs, autoscaler outstanding-request calculations, metrics and the web UI are
all silently wrong. The core also wastes scheduling cycles proposing a node it
believes is empty while the shim rejects every attempt.
{*}Reachability{*}: {{kubectl delete node}} on a live node (or automation doing
the equivalent) followed by kubelet self-re-registration recreates the Node
object within seconds; the pod-GC controller quarantines orphaned pods for ~40s
before deleting them, so pods surviving a brief flap is the normal outcome, not
a race. Cluster-autoscaler is not a trigger (it drains first). Rare but
ordinary operations; when it fires it affects every pod on the node, for an
unbounded duration on long-running workloads.
{*}Reproduction{*}: deterministic in a fault-injection test built on the
MockScheduler harness (bind churn + node remove/re-add through the informer
path): after settle, pods placed in the shim cache have no matching core
allocation (e.g. one flapped node: shim 40 pods, core 0 allocations). 3/3
flapped runs reproduce; 0/2 static controls. Test to be contributed.
{*}Suggested fix{*}: on adoption of a YuniKorn-managed pod whose task is Bound,
re-report the allocation the same way the foreign-pod branch and restart
recovery already do ({{{}UpdateAllocation{}}} with NodeID set -> the core's
{{MarkPreviouslyAllocated}} path). Alternatively (or additionally) reconsider
overloading {{STOPPED_BY_RM}} for core-initiated node-removal releases, so the
shim can distinguish them.
> Node delete/re-add silently corrupts queue accounting: adopted pods are never
> re-reported to the core
> -----------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3375
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3375
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
>
> When a Node object is deleted and re-created while its pods keep running, the
> shim and the core end up permanently disagreeing about what is on that node.
> The shim has the pods placed; the core believes the node is empty and has
> given the queue its quota back. The queue's usage stays understated for as
> long as the pods run.
> *What works (this issue is not about these)*:
> - The orphaned pod tracking works. On node removal the pods are orphaned, and
> when the node comes back the adoption path puts them back on the node in the
> shim cache correctly.
> - There is no node-level double allocation. The shim's own predicates run
> against the shim cache, which contains the adopted pods, so a genuinely full
> node rejects new placements. We verified this.
> *The gap*: adoption only restores the shim's view. Nothing restores the
> core's.
> # On node removal the core removes the allocations from the applications and
> decrements the queue's allocated resources ({{removeNodeAllocations}} ->
> {{queue.DecAllocatedResource}}), then notifies the shim with
> {{TerminationType STOPPED_BY_RM}}. The shim callback correctly only forgets
> the pod in its cache and does not fail the task -- the pod is still running,
> and dispatching the release would delete it. The task stays Bound.
> # On node re-add, the node registration cannot carry allocations
> ({{existingAllocations}} was removed from the SI in
> [YUNIKORN-2780|https://issues.apache.org/jira/browse/YUNIKORN-2780]), and the
> adoption path never sends an allocation update either: {{updateYuniKornPod}}
> finds the still-Bound task and only calls {{SetTaskPod}}.
> # No reconciliation exists. The divergence lasts until each pod terminates,
> or until a scheduler restart (restart recovery does re-report).
> *Why we believe this is an oversight and not a design decision*: the same
> adoption loop DOES send an allocation update to the core for non-YuniKorn
> pods ({{updateForeignPod}} -> {{CreateAllocationForForeignPod}}, tracking
> added by
> [YUNIKORN-2834|https://issues.apache.org/jira/browse/YUNIKORN-2834]). Only
> the YuniKorn-managed branch is silent, because the surviving Bound task
> suppresses it.
> *Impact*:
> - Queue accounting is silently wrong: a queue with max=X can end up running
> more than X (the surviving pods plus a full X of new admissions elsewhere).
> User quotas, fair-share ordering, preemption inputs, autoscaler calculations,
> metrics and the web UI all compute against understated usage.
> - If ALL of an application's allocations were on the removed node, it is
> worse: the core application is left empty, moves Completing -> Completed
> (~30s default), and is removed from the partition entirely. The shim still
> holds the application with Bound tasks; the core no longer knows it exists.
> - The core also wastes scheduling cycles proposing a node it believes is
> empty while the shim rejects every attempt.
> *Reachability*: {{kubectl delete node}} on a live node (or automation doing
> the equivalent) followed by kubelet self-re-registration recreates the Node
> object within seconds. The pod-GC controller quarantines orphaned pods for
> ~40s before deleting them, so the pods surviving a brief flap is the normal
> outcome. Cluster-autoscaler is not a trigger (it drains first). Rare but
> ordinary operations; when it fires it affects every pod on the node.
> *Reproduction*: deterministic via the
> [YUNIKORN-3377|https://issues.apache.org/jira/browse/YUNIKORN-3377]
> fault-injection test ({{TestChaosNodeFlap}}, oracle O5): after a flap, e.g.
> one node shows 40 pods placed in the shim and zero allocations in the core,
> on every seed.
> *Suggested fix*: on adoption of a YuniKorn-managed pod whose task is Bound,
> re-report the allocation the way the foreign-pod branch and restart recovery
> already do ({{UpdateAllocation}} with the node ID set -> the core's
> {{MarkPreviouslyAllocated}} path). Because the core application may already
> have been removed (see impact above), the fix must follow the recovery
> sequence -- re-register the application first, then report the allocations --
> rather than sending the allocation update alone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
