mxm commented on issue #15031: URL: https://github.com/apache/iceberg/issues/15031#issuecomment-3759257687
Session clusters are not the primary deployment model for Flink/Iceberg, but I agree we should address this issue. In the absence of a job shutdown hook, I think the best we can do is to (1) add a shutdown method for ThreadPools, (2) call that method from the Flink Iceberg source / sink on job shutdown. This is a bit fragile and best-effort, but it should work. Alternatively, it could work to start a daemon alongside with the static ThreadPools, which periodically checks whether the associated classloader has already been closed. If it has, it would shut down the thread pools and exit. The drawback of this approach is that Java doesn't directly expose a way to check for closed Classloaders because not all of them are. Flink's URLClassLoader is closable, so with some Java magic it is doable. Finally, I wonder whether depending on static thread pools in Iceberg is a sound design pattern. We could at least add a way to disable creating them in a static way, such that methods like `ThreadPools.getWorkerPool()` can optionally return new thread pools which are managed by the user. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
