stevenzwu commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1402252806
> What is the best way to use Connection pools in Flink tasks? Like a pool for HMSConnection, or JDBC driver? Shall we add that, and the wrapper classes to the main Flink classpath, so the classes are not loaded/reloaded every time? That is a good question. It is a bit challenging. The easier model is share nothing across tasks (e.g. no global static conn pools). Let's say each TM as 8 subtask, each task need to open a client (and connection pool) talking to some external system. Each task is responsible for the ownership and lifecycle of the client. This ties the client/conn pool lifecycle to the Flink subtask lifecycle. Yeah, if it is a global static, I think your suggestion of loading classes in the Flink parent classloader is probably the way to go. But that would requires some meddling of the image packaging of including the connector jars in the `flink/libs` dir. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
