badalprasadsingh opened a new issue, #2629:
URL: https://github.com/apache/iceberg-go/issues/2629

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   Right now, `AddDataFilesByPath(..., ignoreDuplicates=false)` -> checks for 
duplicate paths only against the snapshot the txn started from 
(`planningSnapshot`). when the commit is retried (4 retries by default now), 
the files are added again on top of the new snapshot without checking it. so a 
file that another writer added in the meantime ends up live twice, and its rows 
show up twice. No error is returned.
   
   I mean: with two writers registering the same parquet file:
   
   1. a table with one snapshot; writer A and writer B both load it
   2. both call `AddDataFilesByPath(ctx, []string{".../imported.parquet"}, nil, 
false)`
   3. A commits
   4. B commits: its first attempt fails the branch check, then the retry adds 
the file again and succeeds
   5. the scan gives ids `[0 1 1 2 2]`, expected `[0 1 2]` (B should get an 
error)
   
   with `commit.retry.num-retries=0` (the eralier default), step 4 fails with 
`requirement failed: branch "main" has changed`. a job that is re-run after a 
timeout, while its first run is still committing, hits exactly this.
   
   `AddDataFiles` (pre-built `DataFile`s) has the same plan-time check and the 
same bug. Both happen on v2 and v3 tables. Fast append and merge append 
register no conflict validator, so nothing re-checks the paths on retry.
   <br>
   Lets not let this block v0.8.0 RC, **Status** : will try to pick this up
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to