shyjsarah opened a new pull request, #760: URL: https://github.com/apache/paimon-rust/pull/760
## Summary Support scalar pre-filtering before Top-K for data-evolution/global-index vector search, including DataFusion literal and lateral `vector_search` queries. The scalar predicate is evaluated against the same pinned snapshot as vector search, producing global row IDs that are localized per vector-index shard and passed to the vector backend as an allow-list. A normal Paimon read is used so scalar global indexes such as BTree can prune the filter read while residual evaluation, partial index coverage, and deletion visibility remain exact. ## Changes - Enable `VectorSearchBuilder::with_filter` and `BatchVectorSearchBuilder::with_filter` on the data-evolution path. - Pin scalar filtering, deletion-vector handling, raw fallback, refine, and materialization to one resolved snapshot. - Apply the same global row-ID allow-list to indexed search and unindexed raw fallback. - Localize a shared bitmap once per vector-index shard, skip empty shards, and share bitmaps across batch queries without deep copies. - Add a reusable prepared-filter API so lateral execution evaluates a target-side scalar predicate once per physical plan. - Push fully translated DataFusion predicates into literal `vector_search` while retaining an `Inexact` residual correctness check. - Push fully translated target-only conjuncts into lateral vector search while preserving left-side, cross-side, unsupported, and inexact residual predicates. - Reject NaN predicates from vector pre-filter pushdown because Paimon and Arrow comparison semantics differ. - Show pushed scalar filters in logical and physical `EXPLAIN` output. - Document scalar pre-filter semantics and lateral-query behavior. ## Testing - [x] `cargo fmt --all --check` - [x] `git diff --check` - [x] `cargo test -p paimon --lib table::vector_search_builder::tests` - [x] `cargo test -p paimon-datafusion --lib filter_pushdown::tests::test_vector_prefilter_rejects_nan_literals` - [x] `cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vindex_build_then_vector_search_query -- --nocapture` - [x] `cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_lateral_join_uses_query_vectors -- --nocapture` - [x] `cargo clippy -p paimon --lib --tests -- -D warnings` - [x] `cargo clippy -p paimon-datafusion --lib --tests -- -D warnings` Covered cases include filter-before-Top-K ordering, empty matches, batch reuse, non-zero shard offsets, raw fallback for unindexed rows, time-travel snapshot pinning, literal SQL, lateral SQL, mixed/cross-side conjuncts, and NaN pushdown safety. ## Notes - No on-disk format change. - No SQL function signature change. - Existing `VectorSearch::include_row_ids` public API remains unchanged; this PR adds `PreparedVectorSearchFilter` and shared-filter builder support. - This is scalar pre-filtering for vector search, not the existing vector/full-text route-fusion `hybrid_search` feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
