aylonsk commented on PR #14932:
URL: https://github.com/apache/lucene/pull/14932#issuecomment-3175381978
Thank you to everyone for looking into this issue. While it seems like the
cause is still being determined, I tried to get ahead of the problem and run
some more benchmarks with attention to detail on indexing time, making sure to
reindex every time and use the most updated versions of both implementations. I
didn't run with median results this time either, and will simply post all of
the runs performed.
```
VarInt 1:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.920 1.635 1.608 0.983 100000 100 50 64
250 no 16.10 6209.64 8 41.43
38.147 38.147 HNSW
```
```
GroupVarInt 1:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.922 2.460 2.148 0.873 100000 100 50 64
250 no 25.53 3917.42 8 41.58
38.147 38.147 HNSW
```
Latency improvement: None
index(s) improvement: None
```
VarInt 2:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.920 1.722 1.698 0.986 100000 100 50 64
250 no 15.69 6372.27 8 41.43
38.147 38.147 HNSW
```
```
GroupVarInt 2:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.919 1.664 1.631 0.980 100000 100 50 64
250 no 15.99 6253.91 8 41.59
38.147 38.147 HNSW
```
Latency improvement: ~3.37%
index(s) improvement: None
```
VarInt 3:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.919 1.875 1.712 0.913 100000 100 50 64
250 no 18.91 5287.65 8 41.44
38.147 38.147 HNSW
```
```
GroupVarInt 3:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.919 1.554 1.468 0.945 100000 100 50 64
250 no 15.85 6341.88 8 41.59
38.147 38.147 HNSW
```
Latency Improvement: 17.12%
index(s) Improvement: ~16.18%
```
VarInt 4:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.919 2.043 1.838 0.900 100000 100 50 64
250 no 16.92 5909.47 8 41.44
38.147 38.147 HNSW
```
```
GroupVarInt 4:
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn
beamWidth quantized index(s) index_docs/s num_segments index_size(MB)
vec_disk(MB) vec_RAM(MB) indexType
0.920 1.839 1.754 0.954 100000 100 50 64
250 no 15.87 6302.79 8 41.60
38.147 38.147 HNSW
```
Latency Improvement: ~9.99%
index(s) Improvement: ~6.21%
With the exception of the first run, the improvements seem to have remained
across reindexing. It's possible that there may be some extra cost in the first
run of GroupVInt, but it could also be a result of the variability between
tests. However, it doesn't seem like there is much evidence to support that
this change could've caused such a large drop in indexing throughput.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
