[
https://issues.apache.org/jira/browse/YUNIKORN-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703422#comment-17703422
]
Wilfred Spiegelenburg commented on YUNIKORN-1642:
-------------------------------------------------
I completely agree we cannot let this pass with a WARN. However using a FATAL
will cause the scheduler to be restarted and we go and redo the recovery again.
That does not make sense. The API server is not giving the result back on time.
That could be due to load or just the fact that it will always need more time
than the time outset. If we could be sure that is was load, and thus transient,
a restart would be OK. However we cannot assume that. Even if it is load
related the new list of the pods will put all the load back on again and it
might time out again.
If the listing of pods that has to comes back takes longer then we should wait.
There is not much we can do to tweak the listing of the pods. It only allows
for a split per namespace. Parallel retrieval of the pods per namespace might
be faster but does not solve the issue with one large namespace. So probably
not much gain for a lot of work.
The second option is to remove the time out or set it to a long time, like 1
hour, essentially never time out. Put logging in place around it that shows
every X seconds that the listing is still running and recovery is waiting for
the API server. An administrator can then take action based on that log detail.
That will prevent the scheduler from starting with partial information and
provide feedback to the admin. That is I think a more sustainable option going
forward.
> Scheduler recovery failed due to listing operation timeout
> ----------------------------------------------------------
>
> Key: YUNIKORN-1642
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1642
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Priority: Major
> Labels: pull-request-available
>
> The listing operation in the recovery phase:
> https://github.com/apache/yunikorn-k8shim/blob/c25ac60ffbc175c4966f917da21d184f34dea7b4/pkg/client/apifactory.go#L225.
> This could sometimes fail on some large clusters, the response time from API
> server is not guaranteed. And we see logs like this
> {noformat}
> 2023-03-16T07:00:46.181Z WARN client/apifactory.go:218 Failed
> to sync informers {"error": "timeout waiting for condition"}
> 2023-03-16T07:00:46.182Z INFO general/general.go:344 Pod list
> retrieved from api server {"nr of pods": 0}
> 2023-03-16T07:00:46.182Z INFO general/general.go:365 Application
> recovery statistics {"nr of recoverable apps": 0, "nr of total pods": 0, "nr
> of pods without application metadata": 0, "nr of pods to be recovered": 0}
> I0316 07:00:51.319100 1 trace.go:205] Trace[140954425]: "Reflector
> ListAndWatch"
> name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
> (16-Mar-2023 07:00:16.168) (total time: 35150ms):
> {noformat}
> Since it is a WARN, it continues but the informers did not return anything.
> This confuses the scheduler that nothing needs to be recovered, and it goes
> ahead doing the scheduling. This causes subsequential scheduler failures.
> And eventually, nothing can be scheduled anymore.
> This should be a FATAL error. So the scheduler can be restarted to retry
> recoverying.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
