sakshichitnis27 opened a new pull request, #18625: URL: https://github.com/apache/iceberg/pull/18625
`DynamicWriter` and `DynamicWriteResultAggregator` cache a task writer factory / manifest output file factory that holds the `Table` loaded when the entry was created, and the entries are never reloaded. If a table is dropped and re-created with the same name, later writes can still go to the dropped table's location. This PR clears those caches once `dynamic-sink.cache-refresh-ms` has elapsed, the same interval `TableMetadataCache` already uses, so the re-created table is loaded on the next write. A table re-created within one interval of the last cache load can still hit the old entry. Including the table UUID in the cache keys would close that gap, but it changes serialized classes, so I left it out of this first version. This only touches the Flink 2.3 module for now; happy to backport to 1.20 and 2.2 once the approach looks right. Fixes #18623 Test plan: - `TestDynamicWriter.writesToRecreatedTable` and `TestDynamicWriteResultAggregator.writesManifestsToRecreatedTable` drop and re-create a table with a different data / manifest location and check where the new files land. Both failed before the change. - `TestDynamicWriter.reusesTaskWriterFactoryWithinCacheRefreshInterval` checks the writer still reuses its factory within the interval. - `./gradlew :iceberg-flink:iceberg-flink-2.3:test --tests "org.apache.iceberg.flink.sink.dynamic.*"`, `checkstyleMain`, `checkstyleTest`, `spotlessCheck`. --- **AI Disclosure** - Model: Claude Sonnet 5.5 - Platform/Tool: Claude Code - Human Oversight: [unknown - human to fill in] - Prompt Summary: Reproduce and fix #18623 (stale table cached in DynamicWriter and DynamicWriteResultAggregator after a same-name drop and re-create), with regression tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
