JunRuiLee opened a new pull request, #104: URL: https://github.com/apache/paimon-vector-index/pull/104
## Summary Implements the IVF-RQ step of #97 on top of #99 (`75e391d`). - Add L2 distance range search to both `VectorIndexReader` and `IVFRQIndexReader`, covering single/batch queries with and without a serialized Roaring allow-list. - Reuse `Collector`, `RangeCollector`, `DistanceBand`, `VectorRangeSearchParams`, and `RangeSearchResult`. - Decide membership using the one-bit estimate or the full multi-bit estimate, with half-open bounds, no clamping, no fixed-K truncation, and no ordering guarantee. - Read each unique non-empty probed list once per call, share its payload across queries, and parallelize independent list/query scans. Return result-owned statistics without changing the last top-K statistics. - Update the Rust/core and site documentation with estimator semantics, validation, filtering, and statistics. ## Correctness and compatibility - Range scanning evaluates all stored bit planes using F32 lookup sums. It does not apply top-K's coarse lower-bound or quantized FastScan pruning to estimate-band membership. - The range-only probe path checks every direct query-centroid distance before selection, including distances to unselected lists. Selected distances are reused in estimation, and ties are resolved by list ID. - Non-finite centroids, consumed estimate factors, and computed distances return `InvalidData`. Invalid queries, widths, metrics, and filters retain the shared validation contract. - Probing every list removes the IVF coverage gap, not quantization-induced missing or extra rows relative to an exact-distance predicate. - Existing IVF-RQ top-K scheduling and scan code are unchanged. There are no storage-format changes. This is a focused implementation on current main, not a wholesale transplant of the old `range-search-pr2`. Auto width, limits/caps, benchmark/CI tooling, other families/metrics, and C/JNI bindings are out of scope. ## Tests Adds 13 tests covering an independently accumulated estimated-distance oracle (1–8 bits, padded dimensions and tail blocks), exact boundary membership, single/batch/filter equivalence, signed and large row IDs, fixed probe width, statistics and read capabilities, parallel scans, top-K compatibility, and non-finite data/arithmetic. - `cargo fmt --all -- --check` — passed - `git diff --check` — passed - `cargo test -p paimon-vindex-core --lib --quiet` — 544 passed, 1 existing ignored - `cargo test -p paimon-vindex-core --test range_search --quiet` — 41 passed - Local links and anchors in the four changed HTML pages — passed ## Review Completed local self-review and a separate read-only automated review. Resolved the unselected-centroid overflow validation gap, released excess probe-result capacity after selection, and made the parallel regression test synchronize workers with a bounded wait. No remaining findings in the reviewed scope. GitHub CI is separate from these local results. Rollback is a code revert; no persisted-data migration is required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
