tigerquoll opened a new pull request, #1066: URL: https://github.com/apache/yunikorn-k8shim/pull/1066
### What is this PR for? A failed or aborted bind leaves a stale node assignment in the shim scheduler cache, which later converts the pod into a phantom "existing allocation": queue and node capacity booked to a pod that never ran, and the pod itself permanently Pending. Two cooperating defects in `scheduler_cache.go`: - `forgetPod` passed the cached pod -- which still carries the node name stamped at assume time -- back into `updatePod` before deleting the assumed marker, so the forget re-registered the very assignment it was supposed to drop. - `updatePod` stamped the cached node name onto any incoming pod without an assignment, unconditionally. Informer updates for the (actually unassigned) pod therefore resurrected the stale assignment on every event, and the stamp mutated the pod object owned by the informer cache. The fix: - `forgetPod` reverts the assignment of a pod that is still assumed (clears the node name on a copy) and removes the assumed state before the cache update runs. A pod that is no longer assumed has been bound; its assignment is real and is kept, which also makes double-forget idempotent. - `updatePod` only preserves an existing assignment while the pod is assumed, and stamps it on a `DeepCopy`, never on the caller's object. The new `[YUNIKORN-3316]` AssumePod-retry path also calls `ForgetPod` and inherited the poisoning; fixing the cache heals that caller as well. ### How was this patch tested? - New regression tests: the poisoning sequence (assume, forget, informer update without a node) asserts the pod ends unassigned -- fails on unmodified master, passes with the fix; a companion test pins that a genuinely bound pod keeps its assignment. - Full outage reproduction on a 10,000-pod test rig (apiserver killed for 10s mid bind-burst): master strands 9,659-9,664 of 10,000 pods with ~19,300 phantom "existing allocation" log lines (2/2 runs); with this fix 9,983 of 10,000 bind, 0 phantoms. The 17 residual pods are the separate YUNIKORN-3128 no-retry issue. - The invariant property fuzzer of YUNIKORN-3373 finds this defect on unmodified master on every seed within a minimal three-operation sequence (update-pending, assume, forget); with this fix that invariant holds. - `make test` (race + deadlock detection) green. Generated by the Author with assistance from Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
