mxm commented on code in PR #17437:
URL: https://github.com/apache/iceberg/pull/17437#discussion_r3727454796
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java:
##########
@@ -120,9 +126,18 @@ private class SerializerInfo {
}
private void update() {
- Table table =
catalogLoader.loadCatalog().loadTable(TableIdentifier.parse(tableName));
- schemas = table.schemas();
- specs = table.specs();
+ // The serializer has no teardown hook, so a catalog cannot be held for
reuse; load and
+ // close one per cache miss.
+ try (TableLoader tableLoader =
+ TableLoader.fromCatalog(catalogLoader,
TableIdentifier.parse(tableName))) {
+ tableLoader.open();
+ Table table = tableLoader.loadTable();
+ schemas = table.schemas();
+ specs = table.specs();
+ } catch (IOException e) {
+ // only close() throws IOException here; a failed close should not
fail the lookup
+ LOG.warn("Failed to close catalog for table {}", tableName, e);
+ }
Review Comment:
I'm not sure loading / closing on every update is a good idea. Can we store
`Catalog` in a field? This will get rid of the duplicate loading and only load
once per job.
If we are concerned about not closing one instance, we could close it via
`RuntimeContext#registerUserCodeClassLoaderReleaseHookIfAbsent`, but I'm not
sure this is necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
