dbtsai commented on PR #56377: URL: https://github.com/apache/spark/pull/56377#issuecomment-4654597797
Pushed a follow-up fix (commit `11c8df37cbf`, kept as a separate commit rather than amending). cc @viirya @cloud-fan Closing the residual listener leak in `CleanerCache`: the fast-path I added only avoids registering the `StreamingRunnerCleanerListener` when `isClosing` is already true on entry. In the race where the fast-path check sees `false`, then `close()` starts, then `streamingListener` is initialized and added to `session.streams`, the listener stays registered -- `SessionHolder.close()` does not remove it (it is not tracked in `listenerCache`), so it keeps the `CleanerCache` / `SessionHolder` reachable after close. The post-insert `isClosing` check cleaned the runner but not the listener. Fix: a `listenerRegistered` flag (set when the lazy listener initializes) plus a `removeListenerIfRegistered()` helper that drops the listener from `session.streams` without initializing it as a side effect. It is now called in both: - the closing-session branch of `registerCleanerForQuery` (covers the race), and - `cleanUpAll()` (covers normal close, which otherwise never removed the listener). Added a `cleanUpAll` test asserting the listener is unregistered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
