cloud-fan opened a new pull request, #56140: URL: https://github.com/apache/spark/pull/56140
### What changes were proposed in this pull request? Follow-up to #54106. Two changes to `SparkConnectClient._on_exit`: 1. Bail out immediately if `self._closed` is already `True`, so we do not make any gRPC calls (including the unconditional `_cleanup_ml_cache()`) after the client has been explicitly closed. 2. Run the remaining cleanup work in a daemon thread joined with a 5-second timeout, so process exit is not blocked when the server is unreachable. A daemon thread is abandoned during interpreter shutdown if it does not complete within the timeout. ### Why are the changes needed? `_on_exit` is registered with `atexit` and runs during process shutdown. It unconditionally calls `_cleanup_ml_cache()`, which issues a gRPC `ExecutePlan` call. If the server is unreachable, that call can block for a long time on the gRPC connection timeout before raising — making the Python process appear to hang on exit, both in tests and in production scenarios where the server goes down before the client. The same hang risk applies to `release_session()` and `close()` in the opt-in `SPARK_CONNECT_RELEASE_SESSION_ON_EXIT` branch. See https://github.com/apache/spark/pull/54106#issuecomment-4079259041 for context. ### Does this PR introduce _any_ user-facing change? No. Behavior change is limited to making process exit non-blocking when the Spark Connect server is unreachable. ### How was this patch tested? Existing PySpark Connect tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
