akritishakya opened a new pull request, #2678: URL: https://github.com/apache/iceberg-go/pull/2678
Closes #2629 ## Problem `AddDataFiles` and `AddDataFilesByPath` check for duplicate paths only against the snapshot the transaction was planned on. If another writer commits the same path first, the retry rebuilds the snapshot on the new branch head without checking paths again, so the file is added twice and its rows are read twice. The same happens when `Commit()` is called again after a failed `Commit()` that left the transaction committable. ## Fix - Both functions pass their paths to the append producer, unless the duplicate check is disabled (`WithoutDuplicateCheck()` / `ignoreDuplicates=true`). - When the snapshot is rebuilt on the refreshed branch head, the new `checkDuplicatePaths` (called after `checkRemovedFiles`) fails the commit with `ErrDuplicateDataFiles`, wrapped in `ErrTransactionUnusable`, if any of those paths is a live data file in the new parent. - Docs: the `ErrTransactionUnusable` doc and `concurrent-writes.md` now name both functions for `ErrDuplicateDataFiles`, and `releases.md` has a compatibility note. ## Behavior notes - The check runs only when the snapshot is rebuilt, and reads the new parent's data manifests. - It reads only the new parent, not snapshot history, so a concurrent rollback or snapshot expiry does not break it. - A path whose data file a concurrent commit added and then removed from the table is not live, so the check does not reject it. ## Tests New file `table/add_files_conflict_test.go`. Commit fails with `ErrDuplicateDataFiles`: - `TestAddDataFilesByPath_ConcurrentDuplicateRejected` (v2, v3) - `TestAddDataFiles_ConcurrentDuplicateRejected` - `TestAddDataFilesByPath_RecommitAfterFailedCommitRejected` (the first `Commit` fails with `ErrCommitFailed`, the second with `ErrDuplicateDataFiles`) Commit succeeds with no duplicate rows: - `TestAddDataFilesByPath_ConcurrentDistinctFilesCommit` - `TestAddDataFilesByPath_ConcurrentRollbackCommits` - `TestAddDataFilesByPath_ConcurrentExpiryCommits` - `TestAddDataFilesByPath_ConcurrentAddThenDeleteCommits` Commit succeeds with the duplicate check disabled: - `TestAddDataFilesSkippedDuplicateCheckAllowsConcurrentDuplicate` The three tests that expect `ErrDuplicateDataFiles` fail without this change and pass with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
