[
https://issues.apache.org/jira/browse/YUNIKORN-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YUNIKORN-3377:
-------------------------------------
Labels: pull-request-available (was: )
> Add fault-injection tests with phantom-allocation oracles for the shim
> ----------------------------------------------------------------------
>
> Key: YUNIKORN-3377
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3377
> Project: Apache YuniKorn
> Issue Type: Test
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
> Labels: pull-request-available
>
> Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
> where the fuzzer checks the scheduler cache as a data structure, this drives
> the REAL scheduling stack (Context, Application/Task state machines,
> dispatcher, embedded core through the MockScheduler harness) while injecting
> the faults a real cluster produces, and asserts end-state oracles once the
> cluster settles.
> Two scenarios:
> # *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of
> Bind calls; a kubelet simulator turns each successful bind into
> assigned/Running informer updates, and a resync simulator keeps re-reporting
> never-bound pods as unassigned -- the informer model is truthful to what a
> real cluster does.
> # *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
> delivered through the informer path, modelling a Node object deleted and
> re-created while its pods keep running ({{kubectl delete node}} + kubelet
> self-re-registration, inside the pod-GC ~40s orphan quarantine).
> Oracles (end-state invariants, valid under any interleaving):
> - O1: a pod that never bound successfully has no node assignment in the shim
> cache (the YUNIKORN-3355 phantom oracle)
> - O2: a bound pod is in the cache on exactly the node it was bound to
> - O3: no pod is left assumed once the cluster is idle
> - O4: no pod's assignment points at a node absent from the cache
> - O5: every pod the shim has placed on a node has a matching core allocation
> on that node (shim/core consistency, read directly from the core partition
> state)
> *These tests currently FAIL on master by design*:
> - The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
> (a failed bind leaves a phantom assignment; on master 100% of never-bound
> pods leak one).
> - The node-flap scenario additionally fails via O5 -- a node delete/re-add
> permanently desynchronizes shim and core allocation state and corrupts queue
> accounting: YUNIKORN-3375.
> - Running the scenarios under {{-race}} also surfaced a data race on
> {{Task.pod}}: YUNIKORN-3376.
> The accompanying PR stays a draft until those fixes merge, after which the
> bind scenario passes and the flap scenario passes with the re-report fix; the
> tests then become permanent regression guards.
> *New issues found by these tests*:
> - YUNIKORN-3375: Node delete/re-add silently corrupts queue accounting:
> adopted pods are never re-reported to the core (node-flap scenario, oracle O5)
> - YUNIKORN-3376: Data race on Task.pod: checkPodMetadataBeforeScheduling
> reads without the task lock (any scenario under -race)
> *Pre-existing issues deterministically reproduced*:
> - YUNIKORN-3355: Failed bind leaves a stale node assignment in the shim cache
> (bind scenario, oracle O1)
> - YUNIKORN-3374: Phantom allocation when a node with an assumed pod is
> removed and re-added (node-flap scenario)
> - YUNIKORN-3128: Yunikorn ignores pending pods after apiserver errors
> (mechanism detail added there as a comment: whether a failed bind retries or
> wedges is pure timing)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
