JingsongLi opened a new pull request, #118:
URL: https://github.com/apache/paimon-vector-index/pull/118

   Multi-bit IVF-RQ scans spend substantial time refining extra bit planes, and 
filtered searches previously bypassed FastScan. This change reuses the existing 
SIMD kernels for eligible survivor blocks and improves encoding direction at 
higher dimensions.
   
   ## Changes
   
   - Search two scale initializations with bounded refinement for dimensions >= 
256, retaining the original candidate. Lower-dimensional encoding and one-bit 
behavior are preserved.
   - Use nibble-LUT FastScan for dense extra-plane refinement, compact survivor 
iteration, and filter-first block skipping. Keep the original F32 estimator for 
final scores, with conservative outward-rounded pruning bounds and numerical 
regression coverage.
   - Add deterministic filtered ANN benchmarking with independently computed 
filtered ground truth, plus an offline bit/probe tuner that selects on one 
query split and validates on held-out queries before recommending a 
configuration. Library defaults are unchanged.
   - Document the measurements and storage compatibility checks.
   
   ## Performance
   
   Compared with `b083416` using the project's `ann_bench`: Apple M4 Pro, 48 
GiB, 12 Rayon workers, Rust 1.95 release, warm local APFS pages, full public 
datasets, 1,000 queries, Top-10, 4 bits, nlist=1024, nprobe=64. Values are 
medians of three alternating process pairs using a common benchmark driver and 
locked dependencies.
   
   | Dataset | Recall@10 before → after | Total build before → after | Batch 
QPS local | Batch QPS 2 ms | Batch QPS 20 ms |
   | --- | --- | --- | --- | --- | --- |
   | SIFT1M | .9148 → .9148 | 1.092 → 1.117 s | 2,885 → 3,804 | 2,743 → 3,499 | 
2,496 → 3,319 |
   | GIST1M | .9039 → .9153 | 8.151 → 7.602 s | 396 → 2,626 | 403 → 2,489 | 342 
→ 1,768 |
   | GloVe-100 | .8203 → .8203 | 1.085 → 1.049 s | 2,756 → 3,026 | 2,771 → 
2,821 | 2,592 → 2,674 |
   
   Build-time ranges overlap; these measurements do not establish a consistent 
construction speedup. GIST single-query P95 improves from 6.431 → 2.835 ms 
locally, 8.097 → 5.232 ms at 2 ms, and 25.799 → 22.680 ms at 20 ms. SIFT local 
first-sweep P95 is noisy (1.056 → 3.138 ms); a separate same-file, three-pair 
warm control measured sustained P95 of 1.200 → 1.061 ms. Low-dimensional 
tail-latency gains are less conclusive than throughput.
   
   The 2 ms case injects actual sleeps per read. The 20 ms sequential case 
models CPU/I/O time plus 20 ms per dependent round; its batch case uses actual 
sleeps. These are latency models, not live object-store measurements. Read 
bytes and dependent rounds are unchanged. Full build breakdowns, latency 
tables, filtered results, and tuning trade-offs are in `docs/ivf-rq.html`.
   
   ## Compatibility
   
   The v1 format, 64-byte header, flags, rotation type, factor layout, writer, 
metadata reader, and payload decoder are unchanged. Existing indexes need no 
migration and benefit from scan improvements immediately; the encoder's recall 
improvement requires rebuilding.
   
   Both versions pass the golden fixtures. A cross-version matrix covers 144 
files and 288 reader/file checks across bits 1–8, L2/IP/cosine, dimensions 
70/256/960, signed row IDs, partial blocks, and zero queries. Top-K and full 
score digests match for each file across readers; 37 paired payloads differ, 
exercising newly encoded content. GIST1M cross-reading also confirms old 
readers read new indexes and new readers read old indexes.
   
   ## Validation
   
   - `cargo test --workspace`: 706 passed
   - `cargo test -p paimon-vindex-core --release rq --lib`: 39 passed
   - `python3 -m unittest discover -s tools/tests`: 19 passed
   - `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets`, 
`git diff --check`, and ASF license-header checks passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to