XiaoHongbo-Hope opened a new pull request, #8939:
URL: https://github.com/apache/paimon/pull/8939
### Purpose
`GlobalIndexLiveRowFilter` computes the current live global row ids for
deletion-vector tables, used by vector / full-text global-index search to drop
DV-deleted rows from results. It processed files one at a time: add the file's
row-id range, then subtract that file's deletion vector.
On data-evolution tables a single row-id range can be covered by
**multiple** files — an anchor file carrying a deletion vector plus
partial-column files that do not. A DV-less sibling file's `addRange` then
re-added the rows the anchor's DV had removed, so the search could return
deleted rows. The result is order-dependent (wrong when the DV-less file is
processed after the anchor).
Reproduction (anchor deletes rows 0,1; a partial-column file covers the same
`[0,3]` range with no DV): expected live rows `{2,3}`, actual `{0,1,2,3}` →
Top-K returned the deleted `[0,1]`.
### Fix
Deletion is a property of the row id, not the file. Accumulate all deleted
ids across every file, and subtract once at the end
(`liveRows.andNot(deletedRows)`), so a DV-less sibling can no longer re-add
deleted rows. Order-independent.
Fixes the Java filter and its Python twin
(`pypaimon/table/source/global_index_live_row_filter.py`), which had the
identical per-file bug.
### Tests
Regression to follow (a DV + overlapping partial-column file layout via the
`DataEvolutionTestBase` write pattern, asserting the deleted rows are excluded).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
