tigerquoll opened a new pull request, #1069: URL: https://github.com/apache/yunikorn-k8shim/pull/1069
### What is this PR for? Fault-injection tests for the scheduling stack, companion to the invariant property fuzzer of YUNIKORN-3373 one layer up: where the fuzzer checks the scheduler cache as a data structure, this drives the REAL stack (Context, Application/Task state machines, dispatcher, embedded core through the MockScheduler harness) while injecting the faults a real cluster produces, then asserts end-state oracles once the cluster settles (a 3s stability window -- "first idle instant" races the task re-creation path). Two scenarios: - `TestChaosBindFailure`: the mock kube client fails a seeded fraction (p=0.3) of Bind calls; a kubelet simulator turns successful binds into assigned/Running informer updates, a resync simulator keeps re-reporting never-bound pods as unassigned. The informer model is truthful to a real cluster. - `TestChaosNodeFlap`: the same bind churn plus seeded node remove/re-add cycles through the informer path -- a Node object deleted and re-created while its pods keep running (`kubectl delete node` + kubelet self-re-registration, inside the pod-GC ~40s orphan quarantine). Oracles (end-state invariants, valid under any interleaving): O1 a pod that never bound has no node assignment (the YUNIKORN-3355 phantom oracle) · O2 a bound pod is on the node of its last bind · O3 no pod left assumed once idle · O4 no assignment points at an absent node · O5 every shim-placed pod has a matching core allocation on that node (shim/core consistency, read from the core partition state). ### This PR deliberately fails CI on master | test | failure modes on master (3 seeds) | |---|---| | BindFailure | O1 + O3 on every seed: 47/49/40 of 150 never-bound pods each leak a phantom assignment AND stay assumed forever (the YUNIKORN-3355 poisoning); the cluster never settles | | NodeFlap | O5 on every seed (78-121 pods placed in the shim with no core allocation = YUNIKORN-3375), plus O1/O3 on 2 of 3 seeds via 3355 | Verified on a tree with the cache fixes applied: `TestChaosBindFailure` passes 3/3 repeats flake-free (150/150 bound, max 5 attempts, settles ~14s), and `TestChaosNodeFlap` fails ONLY via O5 (107/142/112 of 150) -- the still-unfixed YUNIKORN-3375 desync, which this test pins deterministically. **Draft until** the YUNIKORN-3355 fix (#1066) and the YUNIKORN-3375 fix merge; the tests then go green and become permanent regression guards. One dependency note: these scenarios under `-race` intermittently surface the pre-existing YUNIKORN-3376 data race (~1 in 9-18 subtest runs), so a `-race` CI lane may flake until that (one-line) fix lands too. Running these scenarios is how YUNIKORN-3375 and YUNIKORN-3376 were found. ### How was this patch tested? It is a test. Determinism: fixed seeds, replayable failure reports (per-oracle counts + first 5 pod dumps). Both red gates (master) and green gates (fixed tree, -count=3) recorded above; zero go-deadlock reports across all runs; `make lint` and `make license-check` clean. Generated by the Author with assistance from Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
