JingsongLi opened a new pull request, #119:
URL: https://github.com/apache/paimon-vector-index/pull/119

   PQ query LUT construction pays small-matrix SGEMM overhead, and transposed 
4-bit scans perform scalar table lookups even when SIMD and selective filters 
can reduce the work. This change accelerates those paths while preserving the 
original-row exact prefix, whole-table quantization and overflow fallback.
   
   - Build query LUTs from a lazy compact transposed codebook, shared with 
encoding.
   - Use NEON/AVX2 table shuffles for the quantized transposed 4-bit suffix, 
widening values before u16 accumulation.
   - Scan only matching 4-bit rows when beneficial. Choose the transposed 
crossover from list length and M, and report the work actually performed in 
batch diagnostics.
   - Add query, serialized-quality and crossover benchmarks, a serial 
full-suite runner, and checked-in reports/CSV/JSON. The storage format is 
unchanged.
   
   ### Performance
   
   Fresh main baseline: `7857f341f57db76ace01606314f25f361ace1f91` (includes RQ 
#118). Apple M4 Pro, release defaults, two threads, standard five-index 
benchmark; six independent processes/version/index in alternating order.
   
   | Index | Sequential QPS, main → candidate | Batch QPS, main → candidate | 
Recall@10 |
   | --- | ---: | ---: | ---: |
   | IVF-PQ | 26,010 → 31,627 (+21.6%) | 149,941 → 232,904 (+55.3%) | 47.54%, 
unchanged |
   | IVF-RQ | 22,916 → 22,891 (−0.1%) | 83,209 → 82,495 (−0.9%) | 90.90%, 
unchanged |
   
   Isolating only the crossover against the same three PQ optimizations with 
the previous fixed threshold, the 870,400-row/768D/M96 filter benchmark 
improves 110.481 → 89.233 ms at 6.25% matching density (−19.2%), and 187.456 → 
90.291 ms at 12.5% (−51.8%). A 256-row/25% reader case improves 184.407 → 
140.660 µs with one thread (−23.7%).
   
   The follow-up contains 119 benchmark processes, including forced-policy 
calibration and diagnostic variants, 329 before/after crossover points, 4/8-bit 
public Fashion-MNIST quality checks and the final five-index comparison. All 
result-signature groups and printed recall comparisons agree. The earlier full 
suite contains another 236 processes on historical base `b083416`; its results 
are kept separately.
   
   An earlier RQ decrease on that historical base does not reproduce after 
updating main; no RQ kernel patch is added, and its historical cause is not 
established. Public 4-bit/1% filtering remains −5.2% sequential and −3.3% batch 
when combining initial and rotated trials; the rotated trials alone are 
approximately unchanged. The report retains both phases and a slower 
unchanged-path microbenchmark rather than claiming every case improves. 
Performance is measured on ARM64; x86-64 is compile-checked here.
   
   - [Current-main RQ verification and isolated crossover 
results](https://github.com/JingsongLi/paimon-vector-index/blob/0bdf27b0ece11a0738b612fce0f5f9d94566cc70/docs/benchmarks/pq-rq-fixes-20261111.md)
   - [Full benchmark matrix and historical 
results](https://github.com/JingsongLi/paimon-vector-index/blob/0bdf27b0ece11a0738b612fce0f5f9d94566cc70/docs/benchmarks/pq-full-20261111.md)
   - [Staged measurements for the three PQ 
optimizations](https://github.com/JingsongLi/paimon-vector-index/blob/0bdf27b0ece11a0738b612fce0f5f9d94566cc70/docs/benchmarks/pq-20261111.md)
   
   ### Validation
   
   - `RAYON_NUM_THREADS=2 cargo test --workspace --release`: 712 passed, 2 
existing ignored tests.
   - `cargo clippy --all-targets --workspace -- -D warnings`.
   - `cargo check -p paimon-vindex-core --target x86_64-apple-darwin`.
   - Formatting, staged whitespace checks and ASF license-header validation 
pass.
   - Regression tests cover f64 LUT references, SIMD tails/overflow, 
exact-prefix score bits, crossover boundaries and diagnostic counters.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to