mccullocht commented on PR #16030:
URL: https://github.com/apache/lucene/pull/16030#issuecomment-5283424875
I'll try to revive this over the next week or two.
> This should work well when incoming vectors are isotropic (all dimensions
behave the same i.e. the histograms of their per-dimension values approximate
little baby gaussians with mean 0) and variance as required to be on unit
sphere at that dimensionality.
IIRC the base quantizer should behave pretty well if the distribution is
uncentered as well (this is what the lower/upper interval are for and why the
dot product is unsigned), but if it's not ~Gaussian I would expect the error
rate to be high.
RE: rotation -- I think we might want this to provide this but it should be
layered just above Lucene. My concern is that if rotation is delegated to the
segments during search costs will be very high -- I would expect O(1-2usecs)
for a heavily optimized FWHT on good hardware, and you would have to multiply
this cost by the number of segments. It'll look fine in a benchmark that's
merged to one segment but poor in practice.
> Does this PR make any effort / at least javadocs to explain that you
should ensure your incoming vectors are isotropic? Or to spot check if they
really seem to be isotropic? luceneutil has all sorts of smell detection
("smelling pipeline" a recent genai model called it!) now to detect all sorts
of problems your otherwise very-opaque-to-humans vectors might have.
This is not documented, but is generally a constraint for most quantizers.
It would be easy to document but I'm not sure what we would do if we detected
your vectors don't quantize well. If we had an auto-quantization setting of
some kind at that point you would just fall back to float32 or float16.
In general I'm annoyed by the amount of code I have to copy here, especially
since all of the flat formats on the read side are just a fixed stride read
index read. Maybe the layer of abstraction should be closer to the vector layer
(`float[] -> byte[]`) which would also be usable for unquantized float/byte
inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
