JunRuiLee opened a new pull request, #533: URL: https://github.com/apache/paimon-rust/pull/533
### Purpose Part of #514. Add scalar `WHERE` residual-filter support to the primary-key vector search read path, mirroring Java `PrimaryKeyVectorRead`'s residual support. A caller can attach a data-column predicate via `VectorSearchBuilder::with_filter`; vector recall is then narrowed so only rows satisfying the predicate are returned, while best-first ordering and Top-K still hold. The predicate is used two ways: - **Pushed into the scan** (`PkVectorScan`), where it prunes whole data files by their column stats (`with_scan_all_files()` is retained so level-0 files still reach the exact fallback). - **Applied per row as a residual** over the surviving files: per-file allowed physical positions are computed once and threaded into the bucket search as an allow-list. The ANN path intersects the allow-list into the reader's include-row-ids and re-verifies each returned hit against it; the exact fallback excludes positions outside the allow-list and skips files with no allowed rows without opening a reader. To answer correctly rather than silently mis-answer, the following all fail loud: a filter off the primary-key vector path, a filter without deletion-vectors-enabled + merge-on-read-disabled, an ANN hit resolving outside the pre-filter, and a residual position past a source file's row count. ### Brief change log - `feat(spec)`: add `CoreOptions::deletion_vectors_merge_on_read` accessor (default `false`), used by the residual guard. - `feat(table)`: residual filter support across the PK-vector read path — bucket-search allow-list (ANN + exact fallback), per-file residual position computation, `PkVectorScan` filter pushdown for file-level stats pruning, `map_ann_results` post-verification of ANN hits against the allow-list, out-of-range residual position fails loud, and skipping exact-reader preload for files with an empty residual set. - `test(table)`: end-to-end coverage of the residual filter on the baseline PK-vector fixture. ### Tests - Unit: `build_live_row_ids` (residual∩active∩¬DV intersection, cross-file offsets, out-of-range fail-loud), `map_ann_results` (rejects a hit outside the residual allow-list), `bucket_search` exact residual (allow-list, absent/empty-entry skip, DV intersection), `should_preload_exact_reader`, `residual_positions_by_file` (file-local positions, empty/all/none, non-active skip, missing `first_row_id`), `PkVectorScan::plan` file-level pruning on real tables (a primary-key-column predicate without deletion vectors, and a non-primary-key-column predicate under deletion vectors + no-merge-on-read), and the fail-loud guards. - Integration: `pk_vector_baseline_test` — unfiltered vs residual searches differ and every residual hit satisfies the predicate. - `cargo test -p paimon` green (1617 lib + integration, 0 failed); `cargo clippy -p paimon --lib --tests -- -D warnings` and `cargo fmt --check` clean. ### API and Format - No on-disk format change. - Public API: `VectorSearchBuilder::with_filter(Predicate)` now consumed by the primary-key vector path. - **Note:** file-level stats pruning reads a data file's `value_stats`. The Rust primary-key writer currently stores column stats in `key_stats` and leaves `value_stats` empty, so file-level pruning fires on Java-written tables (the read target of #514) but not yet on Rust-written primary-key tables. Populating writer-side `value_stats` is a separate, pre-existing gap outside this change; correctness is unaffected either way because the per-row residual still applies over surviving files. ### Documentation - Doc comments on the affected functions; no user-facing docs change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
