RKSPD commented on PR #16709: URL: https://github.com/apache/lucene/pull/16709#issuecomment-7026920057
> @RKSPD It'd be great to push this forwards! This would be a breath of fresh air compared to HNSW heavy options Lucene has today (actually, Lucene also has [`FaissKnnVectorsFormat`](https://github.com/apache/lucene/blob/branch_11x/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissKnnVectorsFormat.java) , which itself has many options via its [cool index factory string](https://github.com/facebookresearch/faiss/wiki/The-index-factory)), a completely different vector engine, quickly pluggable via the `Codec` (`KnnVectorsFormat`) API. It'd give options to Lucene users, and also enhance our testing since we could pick vector search from these two diverse options. > > Also, since this is targeting Lucene sandbox module, where things are allowed to be sandy, we shouldn't expect/aim to perfect/polish this PR so much -- it just needs to be basically functional as a starting point, and enhancements come later: progress not perfection. Hi Mike, thanks for your support! I've updated the PR with some quality of life improvements that hopefully makes SegmentIVF behave better at scale and easier to benchmark. There are a lot of cool features here that I hope I can bring to the broader community. The codec behaves a little differently to HNSW, namely it does a lot more work per query at similar recall, but brings down the latency with sequential access and tightly coupled quantizer design. It also has a simpler way to tradeoff RAM or latency for recall, with either `spillBits` or `nprobe` tuneables. It would be really interesting to see SegmentIVF in more benchmarks for LKNN performance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
