gsmiller commented on PR #16567: URL: https://github.com/apache/lucene/pull/16567#issuecomment-5458541995
Disclaimer: Rikhil and I work together and started a conversation about this earlier today, but I wanted to bring it here so others have an opportunity to join the discussion. A property that really excites me about IVF is that docIDs can be traversed in monotonically increasing order within each cluster since it's fundamentally just a postings list of docs (and it also follows that a disjunction of clusters can produce a union of ordered docIDs, exactly the same as something like `TermInSetQuery`). I find this compelling since it means docs could potentially be evaluated doc-at-a-time along with all other clauses of the query (i.e., filters) instead of first collecting semantic results then post-filtering (or alternatively, creating a semantically-unaware filter bitset that gets pushed down into semantic search). But... I don't think this current codec-approach allows this idea to be leveraged. I'm wondering if we could restructure the approach here to allow for this doc-at-a-time scoring. I'm going to look at the code in more detail soon (so I may have an overly-naive mental model of what's going on), but it seems to me like we could create a new Query/Collector implementations that do the following: * Store cluster IDs as simple terms in a standard inverted field with associated docs in the postings (along with embeddings, or possibly put embeddings in a separate docValues field?). Store a graph of cluster nodes using our existing HNSW codec (or maybe something new if it's warranted)? * On query rewrite (or maybe on scorer pull?), do a centroid search to determine the clusters worth unpacking. This could be our existingl HNSW search or something new if we need it. Then, simply rewrite as a TermInSetQuery for those clusters. * In a collector, do full precision scoring for docs that make it that far before putting them in a topK heap. What I'd love to achieve here is a way to run filters doc-at-a-time before doing expensive full-precision scoring. We fundamentally can't do this with HNSW since it can't visit docs in docID order (same problem we have with the points index, just much more expensive because of vector scoring). I'm positive I'm missing some important bits of IVF and your proposed implementation, but I wanted to float this structure to see what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
