JingsongLi opened a new pull request, #8451: URL: https://github.com/apache/paimon/pull/8451
## Summary This PR makes global index based reads aware of deletion vectors so deleted row IDs are not returned by shard global index scans, vector search, or full-text search. It also preserves deletion file metadata while planning global-index build splits so index builders read only live rows. ## Changes - Add a shared deletion-vector row ID filter that reads snapshot data splits and maps file-relative deleted positions back to global row IDs. - Apply deletion-vector filtering to `GlobalIndexScanner` results, including unindexed row ranges. - Apply live-row filters before vector index topK search and add a final safety filter after reranking for single and batch vector search. - Apply deletion-vector filtering before full-text final topK, including compound query and raw fallback paths. - Preserve aligned deletion files when creating shard indexed splits and when sorted global index build splits are regrouped by row range. - Update Spark global-index topology planning to use the snapshot-aware split builder. ## Testing - [x] `git diff --check` - [x] `mvn -pl paimon-core,paimon-spark/paimon-spark-common -am -DskipTests compile` - [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=VectorSearchBuilderTest#testVectorSearchExcludesDeletionVectorRowsBeforeTopK+testGlobalIndexScannerExcludesDeletionVectorRows+testShardIndexedSplitsPreserveDeletionFiles,FullTextSearchBuilderTest#testFullTextSearchExcludesDeletionVectorRowsBeforeTopK,SortedGlobalIndexBuilderSplitTest#testSplitByContiguousRowRangePreservesDeletionFiles test` ## Notes No migration is required. The existing overloads without a snapshot keep their previous behavior; snapshot-aware planning is used by the Spark global index build path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
