slow-J commented on PR #16631: URL: https://github.com/apache/lucene/pull/16631#issuecomment-5571805205
Thanks @neoremind , I had a look at JFR earlier, I ran again to share just for the rerank=false. I tried to attach but even compressed they are too large. Created flamegraph via `jfrconv --cpu-time -o html` will share that instead. [flamegraph-DIFF-baseline-vs-candidate.html](https://github.com/user-attachments/files/31914812/flamegraph-DIFF-baseline-vs-candidate.html) [flamegraph-candidate-rerankfalse.html](https://github.com/user-attachments/files/31914814/flamegraph-candidate-rerankfalse.html) [flamegraph-baseline-rerankfalse.html](https://github.com/user-attachments/files/31914815/flamegraph-baseline-rerankfalse.html) JFR CPU-time profiles, single iteration of baseline + candidate, INDEX_ONLY, `rerank=false` / 1M docs / 1024d Cohere v3 / `quantizeBits=4`. baseline: 1,021,194 `jdk.CPUTimeSample` events candidate: 961,437 `jdk.CPUTimeSample` events | frame (self time) | baseline | candidate | |---|---:|---:| | `OffHeapScalarQuantizedVectorValues.unpackNibbles` | 41,252 / 4.040% | 0 / 0.000% | | `PanamaVectorUtilSupport.int4Unpack` | 0 / 0.000% | 10,721 / 1.115% | | `VectorUtil.int4Unpack` | 0 / 0.000% | 3 / 0.000% | | `int4DotProductSinglePackedBody` | 306,742 / 30.038% | 295,586 / 30.744% | | `FloatVector.reduceLanesTemplate` | 138,670 / 13.579% | 137,674 / 14.320% | | `IntVector.reduceLanesTemplate` | 132,366 / 12.962% | 130,718 / 13.596% | | `AbstractVector.convert0` | 48,744 / 4.773% | 45,560 / 4.739% | Unpack self time goes from 4.040% to 1.115%. Its a 3.62× reduction but it doesn't vanish entirely. Regarding avoiding the intermediate transformation before the dot product: `bothPacked` already exists. `int4DotProductBothPacked` is in `VectorUtilSupport` with scalar, Panama and native implementation. The lucene99 codec uses it (I think?) but lucene104 doesn't. There is an existing JMH test for singlepacked and bothpacked ran ``` java --module-path lucene/benchmark-jmh/build/benchmarks \ -m org.apache.lucene.benchmark.jmh \ 'VectorUtilBenchmark.binaryHalfByte(DotProductSinglePacked|DotProductBothPacked|Unpack)(Vector|Scalar)$' \ -p size=128,1024,4096 \ -wi 5 -i 8 -f 2 \ -rf json -rff ./mh-bothpacked.json ``` and got | dims | SinglePacked | BothPacked | Δ | |---:|---:|---:|---:| | 128 | 34.457 ± 0.436 | 34.648 ± 0.762 | ~noise | | 1024 | 8.888 ± 0.020 | 7.741 ± 0.003 | −12.9% | | 4096 | 2.476 ± 0.003 | 2.092 ± 0.001 | −15.5% | Bothpacked looks slower according to this test, but I didn't spend too long validating it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
