AynSiddiqui opened a new pull request, #2669: URL: https://github.com/apache/iceberg-go/pull/2669
Fixes #2604 ## What On a table upgraded from v2 to v3, a merge-on-read Delete wrote a deletion vector (DV) with only the new deletes. Once a data file has a DV, readers skip its v2 position delete files, so rows deleted before the upgrade came back. ## How When a v3 Delete gives a data file its first DV, the positions from all of that file's v2 position delete files (matched like scan planning does) are merged into the DV. As in Java's BaseDVFileWriter, only file-scoped position delete files (referenced_data_file is that data file) are removed in the same snapshot. Partition-scoped files that can cover several data files are kept: readers ignore them for files that have a DV and still apply them to the others. Equality deletes are not touched. Removing a delete file makes the commit non-replayable, as for existing DV replacement, so a concurrent commit fails with ErrCommitFailed and the caller retries. Notes: - Matching the v2 deletes reuses scan planning, so a table with inconsistent v2 deletes (for example a delete file whose spec or partition does not match its data file) now makes the Delete fail with ErrInvalidMetadata, as a scan of that table already does. Before, the Delete did not check this. - Each matched position delete file is read in full once per Delete, so memory grows with its number of positions; a large partition-wide v2 file costs more. ## Tests New tests in `table/mor_v3_upgrade_deletes_test.go`: the issue's scenario with a second delete after the upgrade, several position delete files on one data file, a shared position delete file, a partitioned table, a partition spec change before the upgrade, equality deletes kept, a delete that matches no row still writing the seeded DV, and a concurrent commit failure. They fail on main and pass with the fix. `go test ./table/...` and `make lint` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
