JingsongLi opened a new pull request, #1439: URL: https://github.com/apache/paimon-rust/pull/1439
### Purpose Avoid mandatory raw scoring when a DE vector index is available but scalar indexes cannot evaluate the predicate. Retrieve ANN candidates and check scalar values against the same pinned snapshot. ### Brief change log - Add iterative filtering, enabled by default, with a per-query/per-shard budget of `max(original Top-K * factor, required candidate limit)`. The default factor is 100; limits also respect shard size and backend bounds. - Reuse the open vector reader and check only newly encountered row IDs across the batch. Fall back to exact raw scoring for queries that reach the budget without enough matches. - Preserve explicit Raw sources, deletion handling, allow-lists and reranking; cancellation interrupts scalar reads and stops further rounds. - Route SQL and lateral filters through core while preserving snapshot and runtime handling. A distinct filtered task kind makes older workers reject unsupported work. ### Tests - Core library: 4945 passed, 7 ignored. DataFusion library: 427 passed. SQL vector integration tests: 13 passed. - `cargo clippy --locked -p paimon -p paimon-datafusion --all-targets -- -D warnings` and `cargo fmt --all -- --check` passed. - Full libraries passed with `RUST_MIN_STACK=16777216` and `--test-threads=2`. The shared-cache subprocess readiness test passed separately after a timing-sensitive parallel-suite failure. - Regressions cover refill, budget bounds, exact fallback, shard offsets, allow-lists, explicit Raw sources, mixed OR predicates, plan/task transport, cancellation and non-Tokio callers. ### API and Format Adds `vector-index.iterative-filter.enabled` (default `true`) and `vector-index.iterative-filter.max-candidates-factor` (default `100`). Setting enabled to `false` restores immediate raw fallback. Source-plan provenance is additive and defaults conservatively for old plans. No persisted table or index format changes. Results retain the vector backend's approximate recall. ### Documentation Updated SQL filtering behavior and configuration documentation. Performance gains still need validation on representative workloads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
