craigcondit commented on PR #555: URL: https://github.com/apache/yunikorn-k8shim/pull/555#issuecomment-1478608534
> hi @craigcondit > > I agree we should increase the timeout value. It is good now this is configurable. But I think we still need a timeout. If it hangs for a very long time (configurable value), e.g 10 or 15 minutes, if it is still stuck, we should let it fail. A failure can trigger alerts to notify us something is wrong; if it is stuck forever, it will take a long time for people to detect something is wrong. No, this is completely WRONG. If it's taking 10-15 minutes, that probably indicates the API server is throttling, or we have a very large cluster. The worst thing to do would be die uncleanly and restart and go through the whole process again. In the case of API server overload, you're just adding more pain. The underlying informers handle failures gracefully, and retry during sync at a lower level. In fact, they can even tolerate the API server being gone for a very extended time. Imagine a reboot of a server that takes 15 minutes where the API server is running - this (while not common) does happen, and the informers will gracefully pick up where they left off when the API server returns. This will manifest as a longer than normal startup, but will otherwise succeed (if we let it). Nothing will ever be "stuck forever" as you say; any such behavior would indicate a severely broken cluster that cannot be dealt with in a sane fashion. Whatever timeout you choose, there will exist a cluster that just needs a few seconds longer, and now you've created pain where there shouldn't be. If startup time is a concern, add external checks on how long it takes before the scheduler reports healthy via APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
