AynSiddiqui opened a new pull request, #2669:
URL: https://github.com/apache/iceberg-go/pull/2669

   Fixes #2604
   
   ## What
   On a table upgraded from v2 to v3, a merge-on-read Delete wrote a deletion 
vector (DV) with only the new deletes. Once a data file has a DV, readers skip 
its v2 position delete files, so rows deleted before the upgrade came back.
   
   ## How
   When a v3 Delete gives a data file its first DV, the positions from all of 
that file's v2 position delete files (matched like scan planning does) are 
merged into the DV. As in Java's BaseDVFileWriter, only file-scoped position 
delete files (referenced_data_file is that data file) are removed in the same 
snapshot. Partition-scoped files that can cover several data files are kept: 
readers ignore them for files that have a DV and still apply them to the 
others. Equality deletes are not touched. Removing a delete file makes the 
commit non-replayable, as for existing DV replacement, so a concurrent commit 
fails with ErrCommitFailed and the caller retries.
   
   Notes:
   - Matching the v2 deletes reuses scan planning, so a table with inconsistent 
v2 deletes (for example a delete file whose spec or partition does not match 
its data file) now makes the Delete fail with ErrInvalidMetadata, as a scan of 
that table already does. Before, the Delete did not check this.
   - Each matched position delete file is read in full once per Delete, so 
memory grows with its number of positions; a large partition-wide v2 file costs 
more.
   
   ## Tests
   New tests in `table/mor_v3_upgrade_deletes_test.go`: the issue's scenario 
with a second delete after the upgrade, several position delete files on one 
data file, a shared position delete file, a partitioned table, a partition spec 
change before the upgrade, equality deletes kept, a delete that matches no row 
still writing the seeded DV, and a concurrent commit failure. They fail on main 
and pass with the fix. `go test ./table/...` and `make lint` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to