anishshri-db opened a new pull request, #42010: URL: https://github.com/apache/spark/pull/42010
### What changes were proposed in this pull request? Shutdown scheduled executor used for maintenance task if an error is reported ### Why are the changes needed? Without this change, we basically would never clean up the maintenance task threads and the pointer is lost too, so we have no way to shutdown this scheduled executor which could eventually lead to thread exhaustion. The change here fixes this issue. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added unit tests ``` [info] - SPARK-44438: maintenance task should be shutdown on error (759 milliseconds) 13:59:57.755 WARN org.apache.spark.sql.execution.streaming.state.StateStoreSuite: ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.streaming.state.StateStoreSuite, threads: rpc-boss-3-1 (daemon=true), shuffle-boss-6-1 (daemon=true) ===== [info] Run completed in 2 seconds, 255 milliseconds. [info] Total number of tests run: 1 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
