JingsongLi opened a new pull request, #119: URL: https://github.com/apache/paimon-vector-index/pull/119
PQ query LUT construction pays small-matrix SGEMM overhead, and transposed 4-bit scans perform scalar table lookups even when SIMD and selective filters can reduce the work. This change accelerates those paths while preserving the original-row exact prefix, whole-table quantization and overflow fallback. - Build query LUTs from a lazy compact transposed codebook, shared with encoding. - Use NEON/AVX2 table shuffles for the quantized transposed 4-bit suffix, widening values before u16 accumulation. - Scan only matching 4-bit rows when beneficial. Choose the transposed crossover from list length and M, and report the work actually performed in batch diagnostics. - Add query, serialized-quality and crossover benchmarks, a serial full-suite runner, and checked-in reports/CSV/JSON. The storage format is unchanged. ### Performance Fresh main baseline: `7857f341f57db76ace01606314f25f361ace1f91` (includes RQ #118). Apple M4 Pro, release defaults, two threads, standard five-index benchmark; six independent processes/version/index in alternating order. | Index | Sequential QPS, main → candidate | Batch QPS, main → candidate | Recall@10 | | --- | ---: | ---: | ---: | | IVF-PQ | 26,010 → 31,627 (+21.6%) | 149,941 → 232,904 (+55.3%) | 47.54%, unchanged | | IVF-RQ | 22,916 → 22,891 (−0.1%) | 83,209 → 82,495 (−0.9%) | 90.90%, unchanged | Isolating only the crossover against the same three PQ optimizations with the previous fixed threshold, the 870,400-row/768D/M96 filter benchmark improves 110.481 → 89.233 ms at 6.25% matching density (−19.2%), and 187.456 → 90.291 ms at 12.5% (−51.8%). A 256-row/25% reader case improves 184.407 → 140.660 µs with one thread (−23.7%). The follow-up contains 119 benchmark processes, including forced-policy calibration and diagnostic variants, 329 before/after crossover points, 4/8-bit public Fashion-MNIST quality checks and the final five-index comparison. All result-signature groups and printed recall comparisons agree. The earlier full suite contains another 236 processes on historical base `b083416`; its results are kept separately. An earlier RQ decrease on that historical base does not reproduce after updating main; no RQ kernel patch is added, and its historical cause is not established. Public 4-bit/1% filtering remains −5.2% sequential and −3.3% batch when combining initial and rotated trials; the rotated trials alone are approximately unchanged. The report retains both phases and a slower unchanged-path microbenchmark rather than claiming every case improves. Performance is measured on ARM64; x86-64 is compile-checked here. - [Current-main RQ verification and isolated crossover results](https://github.com/JingsongLi/paimon-vector-index/blob/0bdf27b0ece11a0738b612fce0f5f9d94566cc70/docs/benchmarks/pq-rq-fixes-20261111.md) - [Full benchmark matrix and historical results](https://github.com/JingsongLi/paimon-vector-index/blob/0bdf27b0ece11a0738b612fce0f5f9d94566cc70/docs/benchmarks/pq-full-20261111.md) - [Staged measurements for the three PQ optimizations](https://github.com/JingsongLi/paimon-vector-index/blob/0bdf27b0ece11a0738b612fce0f5f9d94566cc70/docs/benchmarks/pq-20261111.md) ### Validation - `RAYON_NUM_THREADS=2 cargo test --workspace --release`: 712 passed, 2 existing ignored tests. - `cargo clippy --all-targets --workspace -- -D warnings`. - `cargo check -p paimon-vindex-core --target x86_64-apple-darwin`. - Formatting, staged whitespace checks and ASF license-header validation pass. - Regression tests cover f64 LUT references, SIMD tails/overflow, exact-prefix score bits, crossover boundaries and diagnostic counters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
