JingsongLi opened a new pull request, #66: URL: https://github.com/apache/paimon-vector-index/pull/66
## What changed - expose block-aggregated IVF-RQ search statistics through the unified Reader and ANN benchmark CSV - reuse the coarse-centroid distances already computed by IVF probing, with resident centroid norms - add an adaptive 16-entry LUT FastScan kernel for wide vectors, using NEON/AVX2 where available and conservative complete-distance bounds - seed single-query Top-K thresholds from a 32-vector prefix before the parallel list scan - document staged public-data A/B results and refresh the IVF-PQ warm-local results after rebasing onto #65 The final IVF-RQ ranking still replays every persisted RQ bit plane in the original scalar order. This is exact in the compressed domain and does not require or claim raw-vector refinement. ## IVF-RQ public benchmark Apple M4 Pro, 12 Rayon workers, warm APFS pages, 1,000 public queries, `nlist=1024`, `nprobe=64`, `k=10`, four RQ bits: | Dataset | P95 | Sequential QPS | Batch QPS | |---|---:|---:|---:| | SIFT1M | 1.101 → 1.081 ms (-1.8%) | 1,020 → 1,042 (+2.1%) | 2,706 → 2,743 (+1.4%) | | GIST1M | 5.599 → 5.190 ms (-8.7%) | 203 → 218 (+8.1%) | 325 → 399 (+20.0%) | | GloVe-100 | 1.071 → 1.046 ms (-2.3%) | 1,041 → 1,057 (+1.6%) | 2,855 → 2,839 (-0.6%, treated as noise) | The wide-vector GIST FastScan kernel alone improved P95 by 8.3%, sequential QPS by 6.6%, and batch QPS by 17.8% versus the optimized scalar scanner. SIFT and GloVe deliberately retain the scalar byte-LUT path below padded dimension 256. Every staged binary returned the same Recall@10; the GIST timing A/B used the public Open VDB mirror, so the docs retain the prior cross-index recall value and use timing deltas only. ## Rebased IVF-PQ validation This PR does not reimplement #65; it refreshes the documentation against that rebased default. In six alternating same-file `Off`/`Auto` runs, median SIFT/GIST/GloVe batch QPS changed from 4,191 / 497 / 4,366 to 7,899 / 950 / 8,048 (+88.5% / +91.3% / +84.3%). File and query bytes were unchanged. GIST and GloVe Recall@10 were unchanged at four decimals; SIFT moved from 0.7143 to 0.7142 because the stable f64 factored-table path is numerically close but not bit-identical to direct residual-table accumulation. ## Validation - `RAYON_NUM_THREADS=4 cargo test --workspace --release` (419 passed, 0 failed, 1 ignored in core; integration, FFI, JNI, and doc tests passed) - `cargo clippy --workspace --all-targets -- -D warnings` - `cargo fmt --check` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
