Dale Richardson created YUNIKORN-3374:
-----------------------------------------
Summary: Phantom allocation when a node with an assumed pod is
removed and re-added
Key: YUNIKORN-3374
URL: https://issues.apache.org/jira/browse/YUNIKORN-3374
Project: Apache YuniKorn
Issue Type: Bug
Components: shim - kubernetes
Reporter: Dale Richardson
When a node is removed while a pod is assumed on it (allocation made, bind not
yet confirmed), the shim scheduler cache orphans the pod with the shim-invented
{{Spec.NodeName}} still set. If the node re-appears (kubelet
restart/re-registration, transient node object deletion and rejoin), the orphan
adoption path re-assigns the never-bound pod to it and re-registers the task,
and task recovery reports an existing allocation to the core for a pod that
never ran: a phantom allocation that books queue and node capacity to nothing.
Found by the invariant property fuzzer of YUNIKORN-3373. Sibling of
YUNIKORN-3355 (same phantom-allocation class, different trigger: a node flap
instead of a failed bind).
*Mechanism* (all verified in code on master):
# {{SchedulerCache.removeNode}} ({{scheduler_cache.go}} ~198) iterates the
NodeInfo's pods and orphans ALL of them: it deletes the {{assignedPods}} and
{{assumedPods}} entries and puts the pod into {{orphanedPods}}. The
assumed/bound distinction is erased, and an assumed pod's cached object keeps
the {{Spec.NodeName}} stamped at assume time -- a value the shim invented,
which the apiserver's pod object never had.
# {{updateNode}} on node (re-)add (~168) adopts any orphan whose
{{Spec.NodeName}} matches the new node via {{updatePod}}: the phantom
assignment is back.
# {{Context.addNode}} ({{context.go}} ~211-218) registers each adopted pod via
{{updateYuniKornPod}} -> {{ensureAppAndTaskCreated}}. The original task is
typically gone by then (the core released the node's allocations when the node
was removed), so {{addTask}} re-creates it with metadata carrying the
stale-NodeName pod, and the recovery path reports the pod as an existing
allocation on the re-added node.
# The shim-cache half self-heals on the next informer update for the pod (it
arrives unassigned), but the core-side allocation does not; it persists until
released.
Note the asymmetry that makes this a bug: for a genuinely BOUND pod,
orphan-and-adopt is correct -- its node name is real and the pod actually runs
on the flapping node. Only the assumed-never-bound case adopts a fiction.
*Reachability*: requires a node delete + re-add around the assume-to-bind
window of any pod. Node object flaps are routine at scale (kubelet
re-registration, cloud node churn, node delete + rejoin), and the
assume-to-bind window is always open somewhere on a busy cluster. In the
YUNIKORN-3373 fuzzer, 50/50 seeds reach the state and observe the re-adoption
(26,442 step observations over 250k steps; the fuzzer flaps nodes far more
often than production, so this measures reachability, not frequency). A
targeted regression test (assume, RemoveNode, UpdateNode, assert unassigned)
fails on the unfixed code.
*Fix* (PR to follow): in {{removeNode}}, treat assumed pods the way the
YUNIKORN-3355 fix treats them in {{forgetPod}}: revert instead of orphan --
drop the assumed/assigned state and re-store the pod unassigned (node name
cleared on a copy). The pod is then simply rescheduled, which is the correct
outcome as its target node is gone. Bound pods keep the existing orphan/adopt
behaviour unchanged. The YUNIKORN-3373 fuzzer's invariants then guard the fix
permanently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
