msokolov commented on PR #1054: URL: https://github.com/apache/lucene/pull/1054#issuecomment-1207262818
I made the needed changes to KnnGraphTester and ran several tests with different datasets. I believe this version of the PR addresses all the previous comments, and I'd like to move ahead and commit this to main branch soon with a fast follow backport to 9.x. Re tests:The "electronics" dataset is a proprietary set of vectors; the GloVe data set is open source. I rescaled the GloVe data and converted to bytes by multiplying each sample by 400 and then clipping to [-128,127]. The results are reasonable; roughly a 10% decrease in indexing times, and similar or a bit smaller reduction in query times in most cases. In a few cases the reduction is much more - eg in `GloVe-M=100-beamWidth=400`. It seems that in this case the index based on 8-bit vectors fit in a single segment, while the baseline index was spread across two segments. Generally the quantized indexes are smaller. The amount of reduction depends a lot on: (1) the dimension of the vectors and (2) the number of graph arcs per document. Basically the more of the index is taken up by the vector data (.vec file) the greater the reduction in size. ## baseline electronics (256 dim) |recall| latency| nDoc| fanout| maxConn| beamWidth| visited| index ms| |---|---|---|---|---|---|---|---| |0.932| 0.37| 10000| 0| 16| 32| 100| 1497| |0.962| 0.48| 10000| 50| 16| 32| 150| 0| |0.959| 0.44| 10000| 0| 16| 64| 100| 2666| |0.980| 0.57| 10000| 50| 16| 64| 150| 0| |0.940| 0.40| 10000| 0| 32| 32| 100| 1465| |0.968| 0.51| 10000| 50| 32| 32| 150| 0| |0.966| 0.46| 10000| 0| 32| 64| 100| 2602| |0.984| 0.61| 10000| 50| 32| 64| 150| 0| |0.901| 0.55| 100000| 0| 16| 32| 100| 21830| |0.933| 0.71| 100000| 50| 16| 32| 150| 0| |0.938| 0.63| 100000| 0| 16| 64| 100| 41997| |0.963| 0.82| 100000| 50| 16| 64| 150| 0| |0.915| 0.57| 100000| 0| 32| 32| 100| 22212| |0.949| 0.80| 100000| 50| 32| 32| 150| 0| |0.952| 0.71| 100000| 0| 32| 64| 100| 45915| |0.973| 0.99| 100000| 50| 32| 64| 150| 0| |0.853| 0.71| 1000000| 0| 16| 32| 100| 337099| |0.894| 0.92| 1000000| 50| 16| 32| 150| 0| |0.910| 0.84| 1000000| 0| 16| 64| 100| 626227| |0.939| 1.08| 1000000| 50| 16| 64| 150| 0| |0.871| 0.85| 1000000| 0| 32| 32| 100| 372921| |0.910| 1.04| 1000000| 50| 32| 32| 150| 0| |0.928| 1.00| 1000000| 0| 32| 64| 100| 713421| |0.956| 1.30| 1000000| 50| 32| 64| 150| 0| ## electronics 256 quantized bytes |recall| latency| nDoc| fanout| maxConn| beamWidth| visited| index ms| |---|---|---|---|---|---|---|---| |0.932| 0.40| 10000| 0| 16| 32| 100| 1653| |0.961| 0.51| 10000| 50| 16| 32| 150| 0| |0.959| 0.45| 10000| 0| 16| 64| 100| 2992| |0.979| 0.58| 10000| 50| 16| 64| 150| 0| |0.940| 0.40| 10000| 0| 32| 32| 100| 1543| |0.968| 0.54| 10000| 50| 32| 32| 150| 0| |0.966| 0.51| 10000| 0| 32| 64| 100| 2928| |0.984| 0.62| 10000| 50| 32| 64| 150| 0| |0.901| 0.50| 100000| 0| 16| 32| 100| 22502| |0.935| 0.69| 100000| 50| 16| 32| 150| 0| |0.938| 0.64| 100000| 0| 16| 64| 100| 40918| |0.963| 0.81| 100000| 50| 16| 64| 150| 0| |0.916| 0.56| 100000| 0| 32| 32| 100| 23693| |0.948| 0.77| 100000| 50| 32| 32| 150| 0| |0.952| 0.73| 100000| 0| 32| 64| 100| 43528| |0.973| 0.92| 100000| 50| 32| 64| 150| 0| |0.854| 0.61| 1000000| 0| 16| 32| 100| 324533| |0.894| 0.80| 1000000| 50| 16| 32| 150| 0| |0.907| 0.71| 1000000| 0| 16| 64| 100| 575319| |0.937| 0.93| 1000000| 50| 16| 64| 150| 0| |0.872| 0.65| 1000000| 0| 32| 32| 100| 341945| |0.909| 0.88| 1000000| 50| 32| 32| 150| 0| |0.929| 0.87| 1000000| 0| 32| 64| 100| 645656| |0.957| 1.10| 1000000| 50| 32| 64| 150| 0| ## Glove 100 baseline |recall| latency| nDoc| fanout| maxConn| beamWidth| visited| index ms| |---|---|---|---|---|---|---|---| |0.963| 0.65| 10000| 0| 100| 200| 100| 7189| |0.993| 0.99| 10000| 100| 100| 200| 200| 0| |0.973| 0.71| 10000| 0| 100| 400| 100| 11174| |0.996| 1.08| 10000| 100| 100| 400| 200| 0| |0.963| 0.67| 10000| 0| 200| 200| 100| 7634| |0.993| 0.98| 10000| 100| 200| 200| 200| 0| |0.972| 0.73| 10000| 0| 200| 400| 100| 11347| |0.996| 1.09| 10000| 100| 200| 400| 200| 0| |0.887| 1.47| 100000| 0| 100| 200| 100| 215773| |0.952| 2.44| 100000| 100| 100| 200| 200| 0| |0.918| 1.80| 100000| 0| 100| 400| 100| 424821| |0.972| 3.09| 100000| 100| 100| 400| 200| 0| |0.887| 1.46| 100000| 0| 200| 200| 100| 208859| |0.952| 2.55| 100000| 100| 200| 200| 200| 0| |0.918| 1.83| 100000| 0| 200| 400| 100| 430869| |0.972| 3.09| 100000| 100| 200| 400| 200| 0| |0.824| 3.15| 1000000| 0| 100| 200| 100| 3946321| |0.893| 5.18| 1000000| 100| 100| 200| 200| 0| |0.859| 3.61| 1000000| 0| 100| 400| 100| 9402514| |0.923| 6.27| 1000000| 100| 100| 400| 200| 0| |0.874| 3.86| 1000000| 0| 200| 200| 100| 3413582| |0.932| 6.88| 1000000| 100| 200| 200| 200| 0| |0.904| 4.95| 1000000| 0| 200| 400| 100| 7758411| |0.957| 8.66| 1000000| 100| 200| 400| 200| 0| ## Glove 100 byte quantized |recall| latency| nDoc| fanout| maxConn| beamWidth| visited| index ms| |---|---|---|---|---|---|---|---| |0.962| 0.70| 10000| 0| 100| 200| 100| 8259| |0.993| 1.06| 10000| 100| 100| 200| 200| 0| |0.973| 0.74| 10000| 0| 100| 400| 100| 12799| |0.996| 1.15| 10000| 100| 100| 400| 200| 0| |0.962| 0.66| 10000| 0| 200| 200| 100| 8331| |0.993| 1.07| 10000| 100| 200| 200| 200| 0| |0.973| 0.75| 10000| 0| 200| 400| 100| 12497| |0.996| 1.15| 10000| 100| 200| 400| 200| 0| |0.887| 1.32| 100000| 0| 100| 200| 100| 169558| |0.952| 2.08| 100000| 100| 100| 200| 200| 0| |0.917| 1.51| 100000| 0| 100| 400| 100| 363433| |0.972| 2.74| 100000| 100| 100| 400| 200| 0| |0.887| 1.24| 100000| 0| 200| 200| 100| 174706| |0.952| 2.12| 100000| 100| 200| 200| 200| 0| |0.917| 1.51| 100000| 0| 200| 400| 100| 366009| |0.972| 2.70| 100000| 100| 200| 400| 200| 0| |0.820| 1.85| 1000000| 0| 100| 200| 100| 3681623| |0.891| 3.37| 1000000| 100| 100| 200| 200| 0| |0.856| 2.48| 1000000| 0| 100| 400| 100| 8957531| |0.921| 4.37| 1000000| 100| 100| 400| 200| 0| |0.873| 3.66| 1000000| 0| 200| 200| 100| 3041750| |0.932| 6.57| 1000000| 100| 200| 200| 200| 0| |0.905| 4.75| 1000000| 0| 200| 400| 100| 7676780| |0.957| 8.66| 1000000| 100| 200| 400| 200| 0| -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
