SteNicholas opened a new pull request, #502:
URL: https://github.com/apache/paimon-cpp/pull/502

   <!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): 
<description> -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #454
   
   <!-- What is the purpose of the change -->
   Deletion vectors were applied only when rows were read, so a vector search 
through readers created
   by `GlobalIndexScan` on a data-evolution table with 
`deletion-vectors.enabled` still ranked deleted
   rows. They could take the top-k slots, and the read then dropped them: a 
search with `limit` k
   returned fewer than k rows, or none, even when enough live rows passed its 
`pre_filter` and
   `predicate`. #448 excluded deleted rows from full-text ranking only.
   
   The fix aligns with Java `AbstractDataEvolutionVectorRead#preFilters` 
(apache/paimon#8459 and its
   follow-ups):
   
   - **`LiveRowGlobalIndexReader::VisitVectorSearch`**: restricts the search of 
each index shard to the
     live rows of its range, combined with the caller's `pre_filter` through 
`ReplacePreFilter`, which
     keeps the `predicate`, distance type and options. The search is passed on 
unchanged when the whole
     shard is live, and still runs when no row of the shard is live, as Java 
does.
   - **`LiveRowGlobalIndexReader::CreateUnion`**: loads live rows on the 
visiting thread before a vector
     search fans out to the shards, as for full-text search, so a load that 
reads manifests on the
     shared executor never waits for it from one of its own threads.
   - **`GlobalIndexLiveRowFilter::LiveRows`**: takes an optional 
`RowRangeIndex`, as Java
     `GlobalIndexLiveRowFilter#liveRows` takes `rowRanges`, and passes it to
     `SnapshotReader::WithRowRangeIndex`. In data-evolution tables only the 
data files whose row ids
     intersect it are planned, so only their deletion vectors are read. The 
table-level full-text read
     passes `nullopt`, as Java `DataEvolutionFullTextRead` passes `null`.
   - **`GlobalIndexScanImpl`**: each index's union gets a supplier that loads, 
at most once, only the
     live rows of the row ranges that index covers, as Java passes 
`indexedRowRanges`. A deletion vector
     of a data file the index does not cover is not read, even if it cannot be 
read. Full-text unions
     use the same supplier; their results do not change, as live rows are only 
used within a shard's
     range. Live rows are now cached per reader instead of per scan.
   
   A known difference from Java is left unchanged: across multiple index 
shards, the C++ vector union
   returns up to `limit` rows from each shard and does not truncate the merged 
result, while Java
   applies `topK(limit)`. The documentation now states this; aligning it is a 
separate change.
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   UT:
   
   - `src/paimon/core/global_index/live_row_global_index_reader_test.cpp`:
     - `TestSearchUnchangedWhenEveryRowIsLive`: a vector search is passed on 
unchanged when every row
       is live, or when only rows outside the shard are deleted (a full-text 
search too).
     - `TestSearchOnlyLiveRowsOfShard`: the pre-filter passes only the live 
rows of the shard, combined
       with a user pre-filter; field name, limit, query, predicate, distance 
type and options are kept;
       a shard ending at `INT64_MAX`.
     - `TestCreateUnionLoadsLiveRowsBeforeSearchingShards`: live rows are 
loaded on the visiting thread
       with a one-thread executor before a vector search fans out, and a shard 
without live rows
       rejects every row.
     - `TestLiveRowsErrorOnlyFailsSearchesThatRankRows`: a failed load fails a 
vector search on a shard
       and on a union before any shard is searched.
   
   IT:
   
   - `test/inte/global_index_test.cpp` (Lumina):
     - `TestLuminaSearchSkipsRowsDeletedByDeletionVectors`: three index shards 
with deleted rows, one of
       them with every row deleted. A scan created before the deletion still 
ranks the deleted rows;
       afterwards no shard ranks one, with no filter, with a pre-filter, and 
with a pre-filter and a tag
       `predicate` together. Searching only the first shard, whose three rows 
nearest the query are
       deleted, returns its three live rows with `limit` 3, as in the issue, 
and reading them returns
       all three.
     - `TestLuminaSearchSkipsUnindexedDeletionVectors`: a search over the index 
of rows [0, 1] succeeds
       when the deletion vector of an unindexed row cannot be read, as Java
       
`VectorSearchBuilderTest#testVectorLiveRowPlanningSkipsUnindexedDeletionVectors`.
   - `test/inte/data_evolution_table_test.cpp`: 
`DataEvolutionDvTableTest.TestGlobalIndexLiveRows` also
     reads the live rows of a row range only.
   
   ### API and Format
   
   <!-- Does this change affect API in include dir or storage format or 
protocol -->
   No change in `include/`. For tables with deletion vectors and row tracking, 
vector searches through
   readers created by `GlobalIndexScan` rank only live rows. No storage format 
change.
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   No new feature. `docs/source/user_guide/global_index.rst` describes how 
deleted rows are excluded
   from vector search, that data-evolution tables read only the deletion 
vectors of the data files the
   index covers, and that a search across multiple index shards returns up to 
`limit` rows from each
   shard.
   
   ### Generative AI tooling
   
   Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to