jad-grepr opened a new pull request, #18356: URL: https://github.com/apache/iceberg/pull/18356
With several writers on a table that has no snapshot yet, `DynamicIcebergSink` can fail the job with `IllegalArgumentException: Ref main already exists`. A writer whose cache predates another writer's creation of the branch loads the table, which now has the branch, and creates it again, and `TableUpdater.findOrCreateBranch` only recovers from `CommitFailedException`. This skips the create when the loaded table already has the branch. `manageSnapshots()` starts from refreshed metadata, so a branch created after the load is still refused. The create therefore also recovers from that refusal when the refreshed table has the branch. Both checks require a branch, so a tag with that name still fails. This is Flink 2.3 only; 1.20 and 2.2 will follow as a backport. Test plan: new `TestTableUpdater` cases cover another writer creating the branch at two points: - after this writer cached the table; - after it loaded the table, before it manages snapshots. Both fail with `Ref main already exists` without this change. A third case checks that a tag with the branch's name still fails the update. Ran `TestTableUpdater`, `TestTableMetadataCache`, `spotlessCheck`, `checkstyleMain` and `checkstyleTest` for Flink 2.3. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: [unknown - human to fill in] - Prompt Summary: Fix the dynamic sink failing a job with "Ref main already exists" when two writers race to create a branch, seen in our CI, and port the fix from our Iceberg fork to Flink 2.3. The code change, its tests and this description are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
