vishnuprakaz commented on code in PR #17437:
URL: https://github.com/apache/iceberg/pull/17437#discussion_r3719780763
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java:
##########
@@ -120,9 +127,22 @@ private class SerializerInfo {
}
private void update() {
- Table table =
catalogLoader.loadCatalog().loadTable(TableIdentifier.parse(tableName));
- schemas = table.schemas();
- specs = table.specs();
+ // The serializer has no teardown hook, so the freshly loaded catalog is
closed here, after
+ // reading the table metadata, to avoid leaking one per cache miss.
+ Catalog catalog = catalogLoader.loadCatalog();
+ try {
+ Table table = catalog.loadTable(TableIdentifier.parse(tableName));
+ schemas = table.schemas();
+ specs = table.specs();
+ } finally {
+ if (catalog instanceof Closeable) {
+ try {
+ ((Closeable) catalog).close();
+ } catch (IOException e) {
+ LOG.warn("Failed to close catalog {}", catalog.name(), e);
+ }
+ }
Review Comment:
On this I looked into having the loader manage and reuse the catalog. the
problem I ran into is that the serializer has no teardown hook, so as far as I
can tell nothing at this layer could ever close a held catalog. Does that match
your thinking that the reuse part belongs in the TaskManager level cache?
While checking, I noticed loadCatalog() has six other call sites in the sink
that don't cloae the catalog either. The operator-level ones do have close()
hooks, would it be worth a similar interim fix for those, or better to leave
them for the TaskManager level cache work?
For the Closeable check here.... would it make sense to switch update() to
TableLoader.fromCatalog() in a try with resources, like FlinkSink and
IcebergSink do for one shot loads?
but the lifecycle (and the Closeable check) would live inside TableLoader
instead of this class.
Happy to make that change if it sounds right : ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
