JunRuiLee opened a new pull request, #107: URL: https://github.com/apache/paimon-vector-index/pull/107
## Summary Part of #97. Add IVF-PQ L2 distance-range search to all four Rust reader entry points: single and batch queries, with and without a serialized Roaring allow-list. - Reuse `DistanceBand`, `VectorRangeSearchParams`, `RangeSearchResult`, and `RangeCollector`. - Support 4-bit and 8-bit PQ, residual/non-residual encoding, and OPQ. - Read each unique non-empty selected list once per call, share it across queries, and stream oversized lists through the existing bounded reader. Propagate collector failures without returning partial results or reading subsequent chunks. - Preserve the existing top-K paths and v1 storage format. ## Membership and validation Membership is half-open `[lower, upper)` in squared-L2 **estimated-distance** space. Both code widths use floating-point ADC: direct query-to-codebook subvector distances, after optional OPQ and coarse residual subtraction, summed in subquantizer order. The range path does not use u8 FastScan tables, precomputed norm identities, top-K truncation, or original-vector reranking. Membership is independent of list size, batch size, and `optimize_for_search`. Range estimates can differ from top-K's distances; probing every list removes the IVF coverage gap, not PQ quantization error. Results are uncapped and unordered. Each filter-eligible row is fully evaluated, so PQ `early_abandoned` is always zero. Non-finite transformed queries, any coarse distance (including unselected centroids), and consumed PQ estimates fail with `InvalidData`. Filtered-out and unprobed rows are not evaluated. Query shape, metric, probe width, and serialized filters are validated before empty-band shortcuts. ## Scope and independence - Based on community `main` at `643b4ed`, which already includes IVF-RQ #104 and IVF-SQ #106. SQ behavior and tests are preserved. - Does not include #92's 4-bit top-K scan scaling fix or #94's serialization allocation fix. `distance.rs` and the PQ writer are unchanged; `io.rs` changes only add fallible propagation to the existing streamed-reader callback while retaining its top-K wrapper. - No other metrics, language bindings, file-format changes, or separate performance optimizations. ## Verification The initial oracle test failed with PQ `Unsupported` before implementation. Twelve PQ regression tests cover both code widths (including rows beyond 200), an independent decoded-code oracle, residual/OPQ modes, exact boundaries and adjacent bands, non-finite data, cancellation and overflow, sparse/dense/empty filters, statistics, batch/single and query-permutation equivalence, optimization invariance, and unchanged top-K results. A synthetic index larger than 64 MiB also checks bounded streaming, shared physical reads, tail-code layout, and immediate collector-error propagation. On macOS/AArch64 with Rust 1.95.0: - `cargo test --offline --workspace`: 642 passed, 2 existing ignored. - `cargo test --offline --release --workspace`: 642 passed, 2 existing ignored. - The range integration suite contains 64 passing tests across the supported families. - `cargo fmt --all -- --check`: passed. - `cargo clippy --offline --all-targets --workspace -- -D warnings`: passed. - `python3 tools/check_license_headers.py`: passed. - `git diff --check`: passed. Linux/x86_64 and full cross-language integration remain for repository CI. There is no storage migration; reverting this additive change does not require rewriting index files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
