soumasish opened a new pull request, #49249: URL: https://github.com/apache/spark/pull/49249
### What changes were proposed in this pull request? This PR introduces a new configurable idle timeout feature for the Spark Connect server. Specifically, it adds a configuration option (`spark.connect.server.idleTimeout`) that allows administrators to specify a duration after which the Spark Connect server will automatically shut down if no new sessions or requests are received. By default, this value is set to 0 (disabled), preserving current behavior. When enabled (i.e., set to a positive value), the Spark Connect service periodically checks for activity based on recorded session access times. If the server remains idle beyond the configured timeout, it triggers a graceful shutdown of the service. ### Why are the changes needed? Currently, the Spark Connect server remains running indefinitely until manually terminated. In managed environments (e.g., Amazon EMR), the lack of a natural stopping condition means the cluster may never be marked as idle and thus never shut down, leading to unnecessary resource consumption and cost. By introducing this idle timeout: 1. Clusters can automatically terminate when no active Spark Connect sessions are ongoing, leading to better resource utilization. 2. Administrators have more control over lifecycle management, improving operational efficiency in hosted or on-demand environments. ### Does this PR introduce _any_ user-facing change? Yes. A new configuration parameter `spark.connect.server.idleTimeout` is introduced. When set to a positive duration (e.g., 10m), the Spark Connect server will monitor inactivity and stop after the specified period with no requests or active sessions. By default, it is disabled (0), meaning behavior remains unchanged unless the administrator explicitly opts in. ### How was this patch tested? WIP ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
