benwtrent commented on issue #15408: URL: https://github.com/apache/lucene/issues/15408#issuecomment-3498616617
@mccullocht I think your fix works well. One thing I noticed and we should consider with the tests, as the residual norm2 increases, the error will increase relative to the quantization bits allowed. The manual testing values picked for the repeatable test are pretty...extreme. Their residual norms are pretty crazy given the vector we care about is very far from the single centroid. The fix you provided works fine for nibbles, int7 and int8 for the test data. However, it doesn't work for single bit as the norm2 is so huge. I am not 100% sure how to address this directly for these unrealistic vectors. But really, anything where the residual norm2 is huge is likely not going to work very well with low bit values. Regardless, I don't think this should block the fix. Mind opening a PR @mccullocht ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
