JingsongLi opened a new pull request, #1439:
URL: https://github.com/apache/paimon-rust/pull/1439

   ### Purpose
   
   Avoid mandatory raw scoring when a DE vector index is available but scalar 
indexes cannot evaluate the predicate. Retrieve ANN candidates and check scalar 
values against the same pinned snapshot.
   
   ### Brief change log
   
   - Add iterative filtering, enabled by default, with a per-query/per-shard 
budget of `max(original Top-K * factor, required candidate limit)`. The default 
factor is 100; limits also respect shard size and backend bounds.
   - Reuse the open vector reader and check only newly encountered row IDs 
across the batch. Fall back to exact raw scoring for queries that reach the 
budget without enough matches.
   - Preserve explicit Raw sources, deletion handling, allow-lists and 
reranking; cancellation interrupts scalar reads and stops further rounds.
   - Route SQL and lateral filters through core while preserving snapshot and 
runtime handling. A distinct filtered task kind makes older workers reject 
unsupported work.
   
   ### Tests
   
   - Core library: 4945 passed, 7 ignored. DataFusion library: 427 passed. SQL 
vector integration tests: 13 passed.
   - `cargo clippy --locked -p paimon -p paimon-datafusion --all-targets -- -D 
warnings` and `cargo fmt --all -- --check` passed.
   - Full libraries passed with `RUST_MIN_STACK=16777216` and 
`--test-threads=2`. The shared-cache subprocess readiness test passed 
separately after a timing-sensitive parallel-suite failure.
   - Regressions cover refill, budget bounds, exact fallback, shard offsets, 
allow-lists, explicit Raw sources, mixed OR predicates, plan/task transport, 
cancellation and non-Tokio callers.
   
   ### API and Format
   
   Adds `vector-index.iterative-filter.enabled` (default `true`) and 
`vector-index.iterative-filter.max-candidates-factor` (default `100`). Setting 
enabled to `false` restores immediate raw fallback. Source-plan provenance is 
additive and defaults conservatively for old plans. No persisted table or index 
format changes. Results retain the vector backend's approximate recall.
   
   ### Documentation
   
   Updated SQL filtering behavior and configuration documentation. Performance 
gains still need validation on representative workloads.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to