JingsongLi opened a new pull request, #118: URL: https://github.com/apache/paimon-vector-index/pull/118
Multi-bit IVF-RQ scans spend substantial time refining extra bit planes, and filtered searches previously bypassed FastScan. This change reuses the existing SIMD kernels for eligible survivor blocks and improves encoding direction at higher dimensions. ## Changes - Search two scale initializations with bounded refinement for dimensions >= 256, retaining the original candidate. Lower-dimensional encoding and one-bit behavior are preserved. - Use nibble-LUT FastScan for dense extra-plane refinement, compact survivor iteration, and filter-first block skipping. Keep the original F32 estimator for final scores, with conservative outward-rounded pruning bounds and numerical regression coverage. - Add deterministic filtered ANN benchmarking with independently computed filtered ground truth, plus an offline bit/probe tuner that selects on one query split and validates on held-out queries before recommending a configuration. Library defaults are unchanged. - Document the measurements and storage compatibility checks. ## Performance Compared with `b083416` using the project's `ann_bench`: Apple M4 Pro, 48 GiB, 12 Rayon workers, Rust 1.95 release, warm local APFS pages, full public datasets, 1,000 queries, Top-10, 4 bits, nlist=1024, nprobe=64. Values are medians of three alternating process pairs using a common benchmark driver and locked dependencies. | Dataset | Recall@10 before → after | Total build before → after | Batch QPS local | Batch QPS 2 ms | Batch QPS 20 ms | | --- | --- | --- | --- | --- | --- | | SIFT1M | .9148 → .9148 | 1.092 → 1.117 s | 2,885 → 3,804 | 2,743 → 3,499 | 2,496 → 3,319 | | GIST1M | .9039 → .9153 | 8.151 → 7.602 s | 396 → 2,626 | 403 → 2,489 | 342 → 1,768 | | GloVe-100 | .8203 → .8203 | 1.085 → 1.049 s | 2,756 → 3,026 | 2,771 → 2,821 | 2,592 → 2,674 | Build-time ranges overlap; these measurements do not establish a consistent construction speedup. GIST single-query P95 improves from 6.431 → 2.835 ms locally, 8.097 → 5.232 ms at 2 ms, and 25.799 → 22.680 ms at 20 ms. SIFT local first-sweep P95 is noisy (1.056 → 3.138 ms); a separate same-file, three-pair warm control measured sustained P95 of 1.200 → 1.061 ms. Low-dimensional tail-latency gains are less conclusive than throughput. The 2 ms case injects actual sleeps per read. The 20 ms sequential case models CPU/I/O time plus 20 ms per dependent round; its batch case uses actual sleeps. These are latency models, not live object-store measurements. Read bytes and dependent rounds are unchanged. Full build breakdowns, latency tables, filtered results, and tuning trade-offs are in `docs/ivf-rq.html`. ## Compatibility The v1 format, 64-byte header, flags, rotation type, factor layout, writer, metadata reader, and payload decoder are unchanged. Existing indexes need no migration and benefit from scan improvements immediately; the encoder's recall improvement requires rebuilding. Both versions pass the golden fixtures. A cross-version matrix covers 144 files and 288 reader/file checks across bits 1–8, L2/IP/cosine, dimensions 70/256/960, signed row IDs, partial blocks, and zero queries. Top-K and full score digests match for each file across readers; 37 paired payloads differ, exercising newly encoded content. GIST1M cross-reading also confirms old readers read new indexes and new readers read old indexes. ## Validation - `cargo test --workspace`: 706 passed - `cargo test -p paimon-vindex-core --release rq --lib`: 39 passed - `python3 -m unittest discover -s tools/tests`: 19 passed - `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets`, `git diff --check`, and ASF license-header checks passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
