[
https://issues.apache.org/jira/browse/YUNIKORN-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YUNIKORN-1642:
-------------------------------------
Labels: pull-request-available (was: )
> Scheduler recovery failed due to listing operation timeout
> ----------------------------------------------------------
>
> Key: YUNIKORN-1642
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1642
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Priority: Major
> Labels: pull-request-available
>
> The listing operation in the recovery phase:
> https://github.com/apache/yunikorn-k8shim/blob/c25ac60ffbc175c4966f917da21d184f34dea7b4/pkg/client/apifactory.go#L225.
> This could sometimes fail on some large clusters, the response time from API
> server is not guaranteed. And we see logs like this
> {noformat}
> 2023-03-16T07:00:46.181Z WARN client/apifactory.go:218 Failed
> to sync informers {"error": "timeout waiting for condition"}
> 2023-03-16T07:00:46.182Z INFO general/general.go:344 Pod list
> retrieved from api server {"nr of pods": 0}
> 2023-03-16T07:00:46.182Z INFO general/general.go:365 Application
> recovery statistics {"nr of recoverable apps": 0, "nr of total pods": 0, "nr
> of pods without application metadata": 0, "nr of pods to be recovered": 0}
> I0316 07:00:51.319100 1 trace.go:205] Trace[140954425]: "Reflector
> ListAndWatch"
> name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
> (16-Mar-2023 07:00:16.168) (total time: 35150ms):
> {noformat}
> Since it is a WARN, it continues but the informers did not return anything.
> This confuses the scheduler that nothing needs to be recovered, and it goes
> ahead doing the scheduling. This causes subsequential scheduler failures.
> And eventually, nothing can be scheduled anymore.
> This should be a FATAL error. So the scheduler can be restarted to retry
> recoverying.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
