Dale Richardson created YUNIKORN-3354:
-----------------------------------------
Summary: Recovery pod ordering is not reproducible despite
intending to be
Key: YUNIKORN-3354
URL: https://issues.apache.org/jira/browse/YUNIKORN-3354
Project: Apache YuniKorn
Issue Type: Bug
Components: shim - kubernetes
Affects Versions: 1.9.0
Reporter: Dale Richardson
{{Context.registerPods}} sorts the recovered pods before replaying them, with
the stated goal of making the resulting queue order reproducible:
{code:go}
// sort pods by creation time so that overall queue ordering is consistent with
prior runs
sort.Slice(pods, func(i, j int) bool {
return pods[i].CreationTimestamp.Unix() < pods[j].CreationTimestamp.Unix()
})
{code}
Three things prevent that from holding:
* The input comes from {{PodInformer.Lister().List()}}, which returns the
informer's items in map iteration order, so the starting order is not
deterministic between runs.
* The sort key is {{CreationTimestamp.Unix()}} - whole seconds. Kubernetes
creationTimestamp has second resolution at source, so every pod created in the
same second collapses into one equal-key group. For a burst - a Job with
parallelism N, a Deployment scale-up - that is the entire set.
* {{sort.Slice}} is not stable, so equal-key elements are left in an arbitrary
order rather than their input order.
Combined, the order within any same-second group is arbitrary and can differ
between runs of the same recovery.
h3. Result
Queue ordering after a scheduler restart is not reproducible, which is what the
comment says it is trying to achieve. The same non-determinism reaches
per-application task dispatch, which map-iterates and sorts on the same
second-resolution creation time.
h3. Notes
Switching to {{sort.SliceStable}} alone does not fix it, because the input
order is already non-deterministic - stability would preserve an order that
carries no information. It needs a deterministic secondary key. Pod UID is the
obvious candidate; namespace and name would also work and has the advantage of
being readable in logs.
Fixing {{registerPods}} alone is not sufficient either. Submission order to the
core is re-derived on every scheduling pass by the shim's {{getTasks}}, which
map-iterates the task map and unstable-sorts on the same second-resolution
creation time. The same secondary key needs to go there, or the stated goal
still is not met.
Out of scope but worth noting so the ticket is not read as a complete
determinism fix: on the core side, applications tied on submission time are
re-ordered by map iteration on each cycle as well. Within an application,
same-priority same-second asks do keep their arrival order, which is what makes
the replay order above matter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
