akritishakya opened a new pull request, #2678:
URL: https://github.com/apache/iceberg-go/pull/2678

   Closes #2629
   
   ## Problem
   
   `AddDataFiles` and `AddDataFilesByPath` check for duplicate paths only 
against the snapshot the transaction was planned on. If another writer commits 
the same path first, the retry rebuilds the snapshot on the new branch head 
without checking paths again, so the file is added twice and its rows are read 
twice. The same happens when `Commit()` is called again after a failed 
`Commit()` that left the transaction committable.
   
   ## Fix
   
   - Both functions pass their paths to the append producer, unless the 
duplicate check is disabled (`WithoutDuplicateCheck()` / 
`ignoreDuplicates=true`).
   - When the snapshot is rebuilt on the refreshed branch head, the new 
`checkDuplicatePaths` (called after `checkRemovedFiles`) fails the commit with 
`ErrDuplicateDataFiles`, wrapped in `ErrTransactionUnusable`, if any of those 
paths is a live data file in the new parent.
   - Docs: the `ErrTransactionUnusable` doc and `concurrent-writes.md` now name 
both functions for `ErrDuplicateDataFiles`, and `releases.md` has a 
compatibility note.
   
   ## Behavior notes
   
   - The check runs only when the snapshot is rebuilt, and reads the new 
parent's data manifests.
   - It reads only the new parent, not snapshot history, so a concurrent 
rollback or snapshot expiry does not break it.
   - A path whose data file a concurrent commit added and then removed from the 
table is not live, so the check does not reject it.
   
   ## Tests
   
   New file `table/add_files_conflict_test.go`.
   
   Commit fails with `ErrDuplicateDataFiles`:
   - `TestAddDataFilesByPath_ConcurrentDuplicateRejected` (v2, v3)
   - `TestAddDataFiles_ConcurrentDuplicateRejected`
   - `TestAddDataFilesByPath_RecommitAfterFailedCommitRejected` (the first 
`Commit` fails with `ErrCommitFailed`, the second with `ErrDuplicateDataFiles`)
   
   Commit succeeds with no duplicate rows:
   - `TestAddDataFilesByPath_ConcurrentDistinctFilesCommit`
   - `TestAddDataFilesByPath_ConcurrentRollbackCommits`
   - `TestAddDataFilesByPath_ConcurrentExpiryCommits`
   - `TestAddDataFilesByPath_ConcurrentAddThenDeleteCommits`
   
   Commit succeeds with the duplicate check disabled:
   - `TestAddDataFilesSkippedDuplicateCheckAllowsConcurrentDuplicate`
   
   The three tests that expect `ErrDuplicateDataFiles` fail without this change 
and pass with it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to