kumarpritam863 commented on PR #18894: URL: https://github.com/apache/iceberg/pull/18894#issuecomment-6821465172
Thanks @pvary for taking a look and the comment. Sharing is the better direction, and Hive already does it: CachedClientPool is a static cache. For the other catalogs it isn't the case today. A RESTSessionCatalog creates its own HTTPClient with a connection pool, auth session, FileIO and a one-thread rest-metrics-reporter pool per instance, and JdbcCatalog creates its own JdbcClientPool. All of these are released only by close(), which is why BaseMetastoreCatalog, RESTCatalog and InMemoryCatalog are Closeable. The static Flink sink already closes its catalog (TableLoader), and so does Kafka Connect (IcebergSinkTask). The dynamic sink operators are the exception: each loads a catalog from the CatalogLoader and never closes it. This only aligns them, and it wouldn't get in the way of pooling later. If you'd rather have the dynamic sink share one catalog per JVM, I'm happy to look at that instead. Which do you prefer? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
