Praveen created YUNIKORN-1596:
---------------------------------
Summary: Pods marked unschedulable when dynamic PVC times out
Key: YUNIKORN-1596
URL: https://issues.apache.org/jira/browse/YUNIKORN-1596
Project: Apache YuniKorn
Issue Type: Bug
Reporter: Praveen
We are seeing a behavior when a scheduled pod requesting for PVC times out, its
marked as unschedulable. There are no retries on such pod and remain in
'pending' state. With pods in pending, autoscaler does not scale down nodes.
This seems similar to issue discussed here:
[https://github.com/kubernetes/autoscaler/issues/3409]
Error from Yunikorn logs :
ERROR cache/context.go:527 Failed to bind pod volumes \{"podName": "<PODNAME>",
"nodeName": "<IP>", "dynamicProvisions": 1, "staticBindings": 0}
...
...
/workspace/pkg/cache/task.go:382
2023-02-20T00:02:22.368Z ERROR cache/task.go:265 task failed \{"appID":
"<APPID>", "taskID": "45981d91-e543-459b-9657-bdc03b57e26f", "reason": "bind
pod volumes failed, name: <NS/PODNAME>, binding volumes: timed out waiting for
the condition”}
{{From Autoscalar logs}}
I0220 20:47:01.775653 1 static_autoscaler.go:502] Scale down status:
unneededOnly=true lastScaleUpTime=2023-02-20 19:20:56.429598603 +0000 UTC
m=+249612.380355315 lastScaleDownDeleteTime=2023-02-20 06:36:50.929515212 +0000
UTC m=+203766.880271921 lastScaleDownFailTime=2023-02-17 22:01:33.693397034
+0000 UTC m=+49.644153730 scaleDownForbidden=true isDeleteInProgress=false
scaleDownInCooldown=true
I0220 20:47:11.787999 1 static_autoscaler.go:228] Starting main loop
I0220 20:47:11.792789 1 filter_out_schedulable.go:65] Filtering out
schedulables
I0220 20:47:11.792953 1 scheduler_binder.go:829] All bound volumes for
Pod "<podname>" match with Node <node>"
I0220 20:47:11.792981 1 filter_out_schedulable.go:118] Pod <podname>
marked as unschedulable can be scheduled on node <node> (based on hinting).
Ignoring in scale up.
# Can Yunikorn introduce retries for such scenarios?
# Can pods be set to error state after retries?
{{Note: pod name, nodename and ip masked above}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
