xi-db opened a new pull request, #46064:
URL: https://github.com/apache/spark/pull/46064

   ([Original PR](https://github.com/apache/spark/pull/46027))
   ### What changes were proposed in this pull request?
   Expired sessions are regularly checked and cleaned up by a maintenance 
thread. However, currently, this process is synchronous. Therefore, in rare 
cases, interrupting the execution thread of a query in a session can take 
hours, causing the entire maintenance process to stall, resulting in a large 
amount of memory not being cleared.
   We address this by introducing asynchronous callbacks for execution cleanup, 
avoiding synchronous joins of execution threads, and preventing the maintenance 
thread from stalling in the above scenarios. To be more specific, instead of 
calling `runner.join()` in `ExecutorHolder.close()`, we set a post-cleanup 
function as the callback through `runner.processOnCompletion`, which will be 
called asynchronously once the execution runner is completed or interrupted. In 
this way, the maintenance thread won't get blocked on joining an execution 
   ### Why are the changes needed?
   In the rare cases mentioned above, performance can be severely affected.
   ### Does this PR introduce _any_ user-facing change?
   ### How was this patch tested?
   Existing tests and a new test `Async cleanup callback gets called after the 
execution is closed` in `SparkConnectServiceE2ESuite.scala`.
   ### Was this patch authored or co-authored using generative AI tooling?

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to