[
https://issues.apache.org/jira/browse/YUNIKORN-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114079#comment-18114079
]
Wilfred Spiegelenburg commented on YUNIKORN-3381:
-------------------------------------------------
This case is not possible: according to K8s definitions a pod cannot transition
from pending to succeeded.
The pod lifecycle definition states:
{quote}Succeeded: The pod reaches this final, terminal phase only after all
containers in the pod have executed and exited cleanly with a exit code of {{0}}
{quote}
Since no containers are running in a pending pod it cannot transition to
succeeded. For the pod to move to running it has to be scheduled and move from
pending to running. Pending to running is a state transition managed by the
kubelet. Moving a pod from running to succeeded is a kubelet managed transition
also. If the node is not registered the kubelet cannot make these changes.
The case you are detailing here would need the following flow:
# pod is created
# pod is notified as new via the informer
# both schedulers see the new pod notification
# 2nd scheduler picks up the pod and schedules it on the node
# YuniKorn gets notified that the node is no longer in the cluster.
# the scheduler action triggers the notification via the informer for the pod
that it is assigned to the node
# the node the pod is scheduled on moves the pod from pending to running
# the pod finishes and the node moves the pod from running to succeeded
Step 5, the node removal, contradicts step 7: the node is no longer in the
cluster and that means the node cannot move the pod into the running state.
That means YuniKorn not only missed the node coming back but also the pod
moving state in step 7. A large breakdown in multiple informers over a longer
period of time.
The root cause could be the notification of the node coming back has not been
received by YuniKorn. It clearly did as step 7 and 8 are required for the pod
to move to succeeded and need the node to be registered.
Step 5, the node removal, cannot happen before step 4 as the 2nd scheduler
would be notified at the same point in time and could not assign the pod to the
node.
> Foreign pod that terminates while orphaned is later adopted as a live
> allocation
> --------------------------------------------------------------------------------
>
> Key: YUNIKORN-3381
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3381
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler, shim - kubernetes
> Reporter: Dale Richardson
> Assignee: Hedger Lai
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
>
> A foreign (non-YuniKorn) pod whose terminal update arrives while its node is
> unknown to the shim is never removed from the scheduler cache. When the node
> later registers, orphan adoption reports the dead pod to the core as a live
> foreign allocation, and it occupies capacity in the shim's predicate cache.
> h3. Mechanism
> {{Context.updateForeignPod}} (pkg/cache/context.go), terminated-release
> branch:
> {code}
> if oldPod != nil && utils.IsPodTerminated(pod) {
> if !ctx.schedulerCache.IsPodOrphaned(string(pod.UID)) {
> ctx.schedulerCache.RemovePod(pod) // + release to the core
> } else {
> // orphaned: "skipping foreign resource update for terminated
> orphaned pod"
> }
> return
> }
> {code}
> Skipping the release for an orphan is correct (nothing is booked in the core
> to
> release). But the branch also skips RemovePod, and it does not update the
> stored
> pod, so the cache keeps the earlier assigned, non-terminal version as an
> orphan.
> When the node arrives, {{updateNodeInternal}} (context.go:212-219) adopts the
> orphans and calls {{updateForeignPod(nil, cachedPod)}} for each foreign one;
> the
> add branch (context.go:432-441) sees an assigned, non-terminated pod and
> books a
> foreign allocation via CreateAllocationForForeignPod. The dead pod is now a
> live
> allocation in the core and an occupying pod in the node's NodeInfo.
> h3. Reproduction
> Three informer events, no scheduling required:
> # add foreign pod assigned to node N (N not yet delivered) -> orphaned, cached
> # update the same pod to Succeeded -> hits the skip branch, stale entry
> retained
> # add node N -> adoption books a foreign allocation for the
> already-terminated pod;
> the shim NodeInfo holds it as an occupying pod (pods=3 observed vs pods=2
> expected)
> Reproduced on master with a direct probe and by an informer event-ordering
> property test, which found it as a convergence divergence (delivering the same
> history with the node first books nothing).
> h3. Impact
> Phantom foreign capacity in both the core and the shim predicate cache after
> any
> window where foreign pods churn while a node is not yet (or temporarily not)
> registered -- node restarts and scheduler startup ordering are the natural
> triggers. Related to the orphan-tracking areas of YUNIKORN-2180 and
> YUNIKORN-3374,
> but a distinct defect.
> h3. Fix sketch
> In the terminated-orphaned branch, remove the pod from the cache (no core
> release
> needed): the entry has no reason to outlive the pod.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
