[
https://issues.apache.org/jira/browse/YUNIKORN-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093353#comment-18093353
]
Wilfred Spiegelenburg commented on YUNIKORN-3128:
-------------------------------------------------
Based on the 2 open PRs (one for core and for for k8shim) here are some
observations:
I have looked in depth at the K8s code for volumes, in general look at these
comment to understand what happens at which stage:
kubernetes/pkg/scheduler/framework/plugins/volumebinding/binder.go line 125
approximately.
We call:
* GetPodVolumeClaims
* FindPodVolumes
* AssumePodVolumes
* BindPodVolumes
The first two calls are made in the assume path as well as the bind path. Based
on the responses from these calls:
{{GetPodVolumeClaims}} : returns two things a list of claims and an error. The
error is always returned as an error and does not look like a retry-able. It
comes back as a error status in the framework. If the list of claims within
that set has unbound claims it is a retry-able state and the pod gets pushed
back for retry. But that only happens on a nil error. The only case I can see
that would be a retry-able error is an API object not found in different ways.
However that should fix itself also when we attempt a new scheduling action.
As it is part of pre-filter the returned {{fwk.Error}} type should remove the
node.
{{FindPodVolumes}} : structure of volume split into static and dynamic,
reasons and an error. An error is not retry-able it even increments the volume
scheduling failed metric. Even without an error but with a non empty reason set
it normally means a failure for volumes, these are the [possible
reasons|https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/framework/plugins/volumebinding/binder.go#L63-L70].
Not something that “fixes” itself on retry. The call is made as part of the
node filter stage, an error {{fwk.Error}} removes the node from the eligible
nodes
{{AssumePodVolumes}} : a boolean and error are returned. When an error is
returned all assumptions are reversed. Not something we should be retrying in
the k8shim. Any failure here should be a reschedule action.
{{BindPodVolumes}} : returns an error only. Internally it does the wait for the
bind timeout. Any error returned here is an automatic retry for scheduling.
I would suggest two changes on the K8s code side:
* Redo the change from YUNIKORN-3123 taking into account the above analysis.
Wrap only the {{GetPodVolumeClaims}} in a retry loop. Also need to carefully
look at the locking of the context (write lock currently), should be able to
fall back on just a read lock but we need to check that.
* Implement scheduler retry for a failed {{context.AssumePod}}
* Implement scheduler retry for any failure in {{context.bindPodVolumes}}
* when that is done we can look at the node {{Bind}} steps and retry.
> Yunikorn ignores pending pods after apiserver errors
> ----------------------------------------------------
>
> Key: YUNIKORN-3128
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3128
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Affects Versions: 1.7.0
> Environment: EKS 1.31
> Reporter: Ruiwen Zhao
> Assignee: Aditya Maheshwari
> Priority: Major
> Labels: pull-request-available
>
> We are running some load testing with Yunikorn, where pods are created at
> 200/s and we monitor if Yunikorn can schedule them at the same rate.
>
> One issue we saw is that Yunikorn ends up ignoring a bunch of (~2000) pods at
> the end of the load test, and completes the application. As shown below,
> there are many pods still Pending, but Yunikorn completes the application
> they belong to, and therefore those pods are stuck. All the pods has
> "schedulerName: yunikorn".
>
> {code:java}
> ❯ kc get pods -n spark8s-kube-burner-yunikorn | grep Pending | head
> kube-burner-0-0-82077 0/1 Pending 0 16m
> kube-burner-0-0-82105 0/1 Pending 0 16m
> kube-burner-0-0-82129 0/1 Pending 0 16m
> kube-burner-0-0-82132 0/1 Pending 0 16m
> kube-burner-0-0-82140 0/1 Pending 0 16m
> kube-burner-0-0-82141 0/1 Pending 0 16m
> 2025-09-29T18:28:18.866Z INFO core.scheduler.fsm
> objects/application_state.go:147 Application state transition {"appID":
> "yunikorn-spark8s-kube-burner-yunikorn-0", "source": "Completing",
> "destination": "Completed", "event": "completeApplication"} {code}
> When looking at one of the Pending pods (kube-burner-0-0-82077), we can
> Yunikorn was trying to schedule it, but failed to do so because of the etcd
> errors. Yunikorn retried once, failed again, and then Yunikorn submitted the
> task again, but no log after that:
> {code:java}
> 2025-09-30T21:18:45.248Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "New",
> "destination": "Pending", "event": "InitTask"}
> 2025-09-30T21:18:45.260Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "Pending",
> "destination": "Scheduling", "event": "SubmitTask"}
> 2025-09-30T21:18:59.464Z ERROR shim.client client/kubeclient.go:127 failed to
> bind pod {"namespace": "spark8s-kube-burner-yunikorn", "podName":
> "kube-burner-0-0-82077", "error": "Operation cannot be fulfilled on
> pods/binding \"kube-burner-0-0-82077\": etcdserver: request timed out"}
> 2025-09-30T21:18:59.465Z ERROR shim.cache.task cache/task.go:464 task failed
> {"appID": "yunikorn-spark8s-kube-burner-yunikorn-0", "taskID":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "reason": "bind pod to node failed,
> name: spark8s-kube-burner-yunikorn/kube-burner-0-0-82077, Operation cannot be
> fulfilled on pods/binding \"kube-burner-0-0-82077\": etcdserver: request
> timed out"}
> 2025-09-30T21:18:59.465Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "Allocated",
> "destination": "Failed", "event": "TaskFail"}
> 2025-09-30T21:18:59.464Z ERROR shim.cache.task cache/task.go:388 bind pod to
> node failed {"taskID": "731dc815-9ee0-4767-a5a9-939219b94f6e", "error":
> "Operation cannot be fulfilled on pods/binding \"kube-burner-0-0-82077\":
> etcdserver: request timed out"}
> 2025-09-30T21:19:25.464Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "New",
> "destination": "Pending", "event": "InitTask"}
> 2025-09-30T21:19:25.464Z INFO shim.fsm cache/task_state.go:381 Task state
> transition {"app": "yunikorn-spark8s-kube-burner-yunikorn-0", "task":
> "731dc815-9ee0-4767-a5a9-939219b94f6e", "taskAlias":
> "spark8s-kube-burner-yunikorn/kube-burner-0-0-82077", "source": "Pending",
> "destination": "Scheduling", "event": "SubmitTask"} {code}
> The failure seems to be caused by etcd timeout, which makes sense, but IMO
> the expected behavior is that Yunikorn keeps trying to schedule the pods with
> backoff.
>
> Yunkorn version: 1.7.0
> Env: EKS 1.31
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
