AnirudhSriram opened a new pull request, #2422:
URL: https://github.com/apache/iceberg-go/pull/2422
Fixes #2421
## Problem
A merge-on-read `Delete` on a v2 table writes position deletes for rows that
earlier position deletes already removed. `makePositionDeleteRecordsForFilter`
builds its scan tasks as `FileScanTask{File, Start, Length}` without
`DeleteFiles`, so `readAllDeleteFiles` finds none and rows that are already
deleted match the filter again.
Query results stay correct, since a duplicate position delete is a no-op for
readers. But delete files grow with every update to a row (each update
re-deletes all older versions of it), and snapshot summary totals
(`total-position-deletes`) are over-counted. v3 deletion vectors are not
affected, because the existing vector is folded into the new one.
## Fix
Before building the tasks, plan the scan with the same row filter
(`t.Scan(...).PlanFiles`), which matches existing delete files to each data
file by sequence number, partition and path. Each task then carries its file's
position delete files, so rows they removed are skipped. Equality delete files
are carried separately and are unchanged.
The cost is one extra scan planning pass per delete.
## Tests
`TestMergeOnReadDeleteSkipsRowsAlreadyDeleted`: appends ids 1, 2, 3, deletes
`id = 1`, then deletes `id IN (1, 2)`. The second delete must add 1 position
delete (total 2). On main it adds 2 (total 3).
`go test ./table/...`, `go vet ./...` and golangci-lint pass.
We hit this on a live table: one update batch wrote 5 position deletes where
3 were correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
