wForget commented on PR #4734: URL: https://github.com/apache/datafusion-comet/pull/4734#issuecomment-4828556073
> The one larger question is whether plugin shutdown() is enough on its own. It only runs on a clean SparkContext.stop(), so a JVM that exits without stopping the context would still hang. Attaching the workers as daemon threads would make the fix robust to that case. That could be a follow-up if you would rather keep this PR focused on the teardown path. Your concern is valid. The plugin `shutdown()` hook is only invoked when `SparkContext.stop()` is called, so it won't help if the JVM exits without stopping the `SparkContext`. `jni-rs` no longer seems to support attaching threads as daemon threads (see https://github.com/jni-rs/jni-rs/issues/469), and since `TOKIO_RUNTIME` is a global object, I haven't found another appropriate lifecycle hook that would reliably trigger its shutdown in that scenario. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
