Hi Everyone, I want to highlight and draw attention to a bug I have recently seen in the Replace Table flow. Today when starting a replace job on a table it's possible to have a scenario where a 2nd commit lands before the replace finishes. This then causes the replace to retry the commit because of a commit conflict exception, but the replace's metadata isn't updated. If the non-replace commit changed the schema then the replace commits metadata is now pointing at column Ids that have moved making it impossible to query data. I have opened a fresh PR that introduces a new unit test for recreating this scenario and a possible solution #16975 <https://github.com/apache/iceberg/pull/16975>
I found issue #16232 <https://github.com/apache/iceberg/issues/16232> that was highlighting this same scenario. I also found two PRs #15904 <https://github.com/apache/iceberg/pull/15904> and #16289 <https://github.com/apache/iceberg/pull/16289>: - #16289 makes concurrent REPLACE fail fast (community −1'd for changing the REPLACE contract). - #15904 rebases the replace onto refreshed metadata (the approach I think is right) but makes broad changes (commit paths, field-ID conflict detection, REST changes) and is still draft with no reviews or updates in the last month. - #16975 (mine) takes #15904's rebase approach: it fixes the history loss and field-ID shift on the existing commit path, and adds the concurrent-drop case. It builds directly on #15904's design. I've added Sreesh (15904's author) as a git co-author with the most recent commit and CI is rerunning, but was green previously. Please let me know if you have any feedback at all, I am just motivated to see this scenario be cleaned up. If opening a third PR wasn't the right approach vs reviewing and highlighting one of the above approaches please let me know here or on the Iceberg slack. I am happy to see any of the PRs get merged. Best Regards, Kurtis C. Wright
