JingsongLi opened a new pull request, #91: URL: https://github.com/apache/paimon-vector-index/pull/91
Sparse IVF training partitions can learn narrow or constant scalar bounds that clip unseen residuals. IVF-SQ also repeats residual allocations, list transposition work, ID decoding, and query heap setup. This change pools residual bounds across the training sample and reduces build and repeated-query overhead while preserving the IVSQ v1 file layout. - Train residual extrema with parallel partition reductions and encode directly into SQ8 output using precomputed scales and NEON/AVX2 conversion. - Transpose serialized lists in parallel within bounded 16 MiB batches. - Reuse one heap per batch query and apply a conservative partial-distance cutoff to L2 blocks. - Cache decoded partitions in a FIFO charged to the existing reader memory budget, including resident metadata and cache bookkeeping. Zero budget disables retention; direct `IVFSQIndexReader::open` remains uncached. - Add behavioral coverage, reproducible Lance/Python benchmark scripts, raw measurements, and updated API, performance, and index-selection documentation. The version, required flags, blocked-code layout, and delta-varint IDs are unchanged. Existing files keep their recorded quantizers. Newly built files store pooled bounds in the existing per-list metadata; the pre-change reader at `8dcabf2` successfully reads them. Rebuilding obtains the new training bounds; reader caching and scan optimizations require the new reader. ### Measured results Three-run medians on Apple M4 Pro, eight workers, 1,024 partitions, 64 probes, Top-10, 65,536 training rows, and 1,000 held-out queries. Values below are **Paimon / Lance 11.0.0**. | Corpus | Index build (s) | Python P95 (ms) | Python batch QPS | Recall@10 | | --- | ---: | ---: | ---: | ---: | | SIFT1M | 0.886 / 3.613 | 0.260 / 1.099 | 9,927 / 4,181 | 0.9812 / 0.9772–0.9775 | | GIST1M | 5.850 / 18.651 | 1.769 / 1.862 | 986 / 1,334 | 0.9399 / 0.9249 | | GloVe-100 | 0.797 / 2.790 | 0.240 / 1.089 | 11,052 / 4,240 | 0.8760 / 0.7843–0.7845 | **GIST batch throughput remains about 26% below Lance.** Its P95 advantage is only 5%, with overlapping three-run latency ranges. Pooled bounds can also lose resolution on extreme-outlier data. Both Python interfaces warm the selected partitions and return IDs/distances without raw-vector refinement. Paimon's reader budget is 4 GiB and Lance's index cache is 1 GiB; each measured index fits both. Lance uses its faster scheduling mode separately for P95 (8) and batch QPS (0). Both engines use native training algorithms with matching sample counts. Build timing includes training through index serialization and excludes Lance source-dataset writing. Native `ann_bench` batch timing includes first-batch reads and cache insertion, so the documentation keeps it separate from warmed Python results. See [the full report](https://github.com/JingsongLi/paimon-vector-index/blob/codex/ivfsq-performance/docs/ivf-sq-performance.md) for baseline comparisons, raw runs, and reproduction commands. ### Validation - `cargo test --workspace`: 510 passed, 2 intentionally ignored, including v1 golden fixtures. - Python bindings: 28 passed. - Format, workspace Clippy with warnings denied, license headers, documentation links/metrics, and benchmark-script syntax checks passed. - The pre-change reader opened newly generated SIFT, GIST, and GloVe files and completed 1,000 single queries plus batch search per corpus; recall differed from the new reader by at most 0.0002. - x86_64 build and 33 SQ tests passed under Rosetta. AVX2 compiled, but Rosetta did not expose AVX2/FMA; native x86 runtime coverage remains for CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
