benwtrent commented on PR #16092:
URL: https://github.com/apache/lucene/pull/16092#issuecomment-4507911000
Its a nice idea. I think we should strive to have a general "precondition
vectors" interface. I am sure on the idea of having it integrated via field
infos...I need to do some thinking here.
Two big issues besides the API that are bothering me:
- Keeping a bunch of rotation matrices on heap for every segment is
unnecessarily expensive
- One precondition for all segments is critical.
I don't know of another API in Lucene that has lazy state cached that is
global over segments...this would be a fairly new thing here. Maybe we can
"hack it" and add something to the KnnFormat reader interface, e.g.
"globalPreconditioning" or something that queries can iterate and
utilize...ugh, but then queries don't have the very nice API of just "search",
and need to do this other step of "precondition"... we don't want to make
things super complex for ALL other vector queries that Lucene has or that
others wrote :(
this is a tough one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
