SteNicholas opened a new pull request, #502:
URL: https://github.com/apache/paimon-cpp/pull/502
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #454
<!-- What is the purpose of the change -->
Deletion vectors were applied only when rows were read, so a vector search
through readers created
by `GlobalIndexScan` on a data-evolution table with
`deletion-vectors.enabled` still ranked deleted
rows. They could take the top-k slots, and the read then dropped them: a
search with `limit` k
returned fewer than k rows, or none, even when enough live rows passed its
`pre_filter` and
`predicate`. #448 excluded deleted rows from full-text ranking only.
The fix aligns with Java `AbstractDataEvolutionVectorRead#preFilters`
(apache/paimon#8459 and its
follow-ups):
- **`LiveRowGlobalIndexReader::VisitVectorSearch`**: restricts the search of
each index shard to the
live rows of its range, combined with the caller's `pre_filter` through
`ReplacePreFilter`, which
keeps the `predicate`, distance type and options. The search is passed on
unchanged when the whole
shard is live, and still runs when no row of the shard is live, as Java
does.
- **`LiveRowGlobalIndexReader::CreateUnion`**: loads live rows on the
visiting thread before a vector
search fans out to the shards, as for full-text search, so a load that
reads manifests on the
shared executor never waits for it from one of its own threads.
- **`GlobalIndexLiveRowFilter::LiveRows`**: takes an optional
`RowRangeIndex`, as Java
`GlobalIndexLiveRowFilter#liveRows` takes `rowRanges`, and passes it to
`SnapshotReader::WithRowRangeIndex`. In data-evolution tables only the
data files whose row ids
intersect it are planned, so only their deletion vectors are read. The
table-level full-text read
passes `nullopt`, as Java `DataEvolutionFullTextRead` passes `null`.
- **`GlobalIndexScanImpl`**: each index's union gets a supplier that loads,
at most once, only the
live rows of the row ranges that index covers, as Java passes
`indexedRowRanges`. A deletion vector
of a data file the index does not cover is not read, even if it cannot be
read. Full-text unions
use the same supplier; their results do not change, as live rows are only
used within a shard's
range. Live rows are now cached per reader instead of per scan.
A known difference from Java is left unchanged: across multiple index
shards, the C++ vector union
returns up to `limit` rows from each shard and does not truncate the merged
result, while Java
applies `topK(limit)`. The documentation now states this; aligning it is a
separate change.
### Tests
<!-- List UT and IT cases to verify this change -->
UT:
- `src/paimon/core/global_index/live_row_global_index_reader_test.cpp`:
- `TestSearchUnchangedWhenEveryRowIsLive`: a vector search is passed on
unchanged when every row
is live, or when only rows outside the shard are deleted (a full-text
search too).
- `TestSearchOnlyLiveRowsOfShard`: the pre-filter passes only the live
rows of the shard, combined
with a user pre-filter; field name, limit, query, predicate, distance
type and options are kept;
a shard ending at `INT64_MAX`.
- `TestCreateUnionLoadsLiveRowsBeforeSearchingShards`: live rows are
loaded on the visiting thread
with a one-thread executor before a vector search fans out, and a shard
without live rows
rejects every row.
- `TestLiveRowsErrorOnlyFailsSearchesThatRankRows`: a failed load fails a
vector search on a shard
and on a union before any shard is searched.
IT:
- `test/inte/global_index_test.cpp` (Lumina):
- `TestLuminaSearchSkipsRowsDeletedByDeletionVectors`: three index shards
with deleted rows, one of
them with every row deleted. A scan created before the deletion still
ranks the deleted rows;
afterwards no shard ranks one, with no filter, with a pre-filter, and
with a pre-filter and a tag
`predicate` together. Searching only the first shard, whose three rows
nearest the query are
deleted, returns its three live rows with `limit` 3, as in the issue,
and reading them returns
all three.
- `TestLuminaSearchSkipsUnindexedDeletionVectors`: a search over the index
of rows [0, 1] succeeds
when the deletion vector of an unindexed row cannot be read, as Java
`VectorSearchBuilderTest#testVectorLiveRowPlanningSkipsUnindexedDeletionVectors`.
- `test/inte/data_evolution_table_test.cpp`:
`DataEvolutionDvTableTest.TestGlobalIndexLiveRows` also
reads the live rows of a row range only.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No change in `include/`. For tables with deletion vectors and row tracking,
vector searches through
readers created by `GlobalIndexScan` rank only live rows. No storage format
change.
### Documentation
<!-- Does this change introduce a new feature -->
No new feature. `docs/source/user_guide/global_index.rst` describes how
deleted rows are excluded
from vector search, that data-evolution tables read only the deletion
vectors of the data files the
index covers, and that a search across multiple index shards returns up to
`limit` rows from each
shard.
### Generative AI tooling
Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]