afterincomparableyum opened a new issue, #7776: URL: https://github.com/apache/kyuubi/issues/7776
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug ServiceDiscovery.stopGracefully loops while sessionManager.getActiveUserSessionCount > 0. If the engine backend has already stopped (e.g. SparkSQLEngine stopped after its SparkContext ended), SessionManager.stop() has shut down the idle-session timeout checker. Sessions left by disconnected clients (common with kyuubi.session.close.on.disconnect=false) are then never closed, the count never drops, and the loop never ends. With etcd, stopGracefully runs on the non-daemon deregister-watcher-thread, so the JVM cannot exit after main returns. On Kubernetes the driver pod stays Running with no SparkContext and no executors until it is deleted by hand. Observed: an engine lost its etcd lease during a long GC pause. Its SparkContext and backend later stopped, and the driver then logged N connection(s) are active, delay shutdown every 10 seconds for hours. KYUUBI #6790 made the loop re-read the count each iteration, but that does not help when nothing can close the sessions. ### Affects Version(s) 1.12.0 ### Kyuubi Server Log Output ```logtalk ``` ### Kyuubi Engine Log Output ```logtalk Kyuubi Engine Log Output INFO KyuubiServiceDiscovery: 15 connection(s) are active, delay shutdown INFO KyuubiServiceDiscovery: 15 connection(s) are active, delay shutdown ... (repeats every 10s indefinitely) ``` ### Kyuubi Server Configurations ```yaml ``` ### Kyuubi Engine Configurations ```yaml Kyuubi Engine Configurations kyuubi.ha.client.class=org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient kyuubi.session.close.on.disconnect=false ``` ### Additional context _No response_ ### Are you willing to submit PR? - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
