First, sorry I didn't write back - the dev mailing list didn't make it to my inbox.
Your understanding is correct.. I will back this more with a full array of tests - which I've done already but would also like to get the dev community's take on how I can test it. I plan to test against 250GB of data applying various values of K and shards with minilm embeddings. One thing I was able to show is that it doesn't affect the speed of running lucene nor did it affect recall from stock lucene OOTB. I've also shown that there are some predictable scenarios where distribution algorithms can greatly help latency and pruning. I created a distributed lucene search engine to show this and so far have been able to show it works in various configurations. https://github.com/ai-pipestream/distributed-search I've found that pruning helps more with higher latency systems. It helps the most if there's one "fast" node (I would call the "scout") and another set of high latency nodes. This is actually good news - you can mix one "fast" node with "slow" ones, which can help in distribution scenarios like a mesh, cell phone, or a raspberry pi. I demonstrated this by doing exactly that - but to a bit of an extreme: I used a large 100GB index spread between 7 shards - 6 raspberry pi nodes and a 32 core AMD server with 3x pci5 nvme raid 0, which provided a 20GB per second throughput. What happened was that the raspberry pi nodes returned back over 50-70% faster with around the same amount of pruning and there was 0 difference in top-k between lucene and the nodes. However, for low values of K, below 100, it didn't make a huge difference and at times causes a slight 3-5% tax. I honestly would love to test this on a full array of fast machines, I'm pretty sure I'd find higher pruning with more coordinated nodes. The biggest win is that I think that this could be, in theory, is when you have a large fleet of fast machines with evenly distributed shards. I have another branch where I was able to tune the coordination up further by using a single UDP socket between nodes to share scores and results were shared with streaming between gRPC nodes. It did help a lot and not reflected in the distributed-search branch. If you have a set of tests I should need to do to accept this, let me know what that should be. On 2026/08/16 19:38:04 Michael Sokolov wrote: > Just to follow up on that -- the general idea of using statistical > estimates of the distribution of results among shards to reduce top-k > sent to worker nodes make lots of sense (I guess followed up by a > retry on select nodes if needed?) -- fully in support of this general > idea. However, if that's what this is about, including classes to do > that in Lucene cannot really present a complete solution since Lucene > doesn't make requests in a distributed system; therefore (1) testing > is best-effort simulation, (2) most distributed systems probably > already have their own ways of doing this. I guess we can provide > reference classes, but I think past attempts to do things like this > (see replication module) haven't really evolved in a healthy way since > they don't get used by the major distributed systems. Have you > considered contributing the solution to Solr, for example? Do you know > how OpenSearch and Elastic handle this? Would they be potential > consumers? > > On Sun, Aug 16, 2026 at 12:58 PM Michael Sokolov <[email protected]> wrote: > > > > I guess what I'm missing here is why cross-node coordination belongs > > in Lucene, which generally only concerns itself with operations on a > > single node. Maybe I haven't properly understood the proposal? My > > understanding is that the new classes would operate in a > > coordinator/collector/collator node that is responsible for calling > > services running on shards and then combining the results: is that > > right? > > > > On Sat, Aug 15, 2026 at 8:02 PM Kristian Rickert <[email protected]> > > wrote: > > > > > > Hi lucene devs, > > > > > > With the advice of some lucene devs, I've been developing a shared-floor > > > kNN collection for Lucene. Having a shared floor allows searches to have > > > a collaborative results live over multiple shards. I've created a PoC > > > search so a gRPC bidi coordinator stream can judge if it has collected > > > the correct number of K values, rather than requiring all K values in an > > > HNSW search as standard distributed search engines demand. > > > > > > The surface area is small, and tests show no impact on baseline > > > performance or recall. After developing and testing this for nearly a > > > year, I have seen latency decrease by over 50% in high-latency > > > environments for large values of K. For values of K < 100, the results > > > vary but not any better or worse in most setups. In low latency > > > environments (fast machines), the feature provides minimal impact for low > > > values of K (under 100). More testing is needed to see if multiple > > > machines could add value. Since I only have 2 fast machines so I can't > > > test distributed shards with more. > > > > > > On a fleet of raspberry pis, it helps tremendously with multiple shards. > > > > > > I am still tweaking the implementation and I'm preparing for a > > > large-scale test using over 500GB of text across 8 shards (I've tested > > > 100GB with success). However, I would like to get a review at this stage > > > to understand what else would be required to land this feature on the > > > mainline. Or if someone has a fleet of fast machines, we can test a PoC > > > search engine that collaborates lucene results. > > > > > > Also, I'd love some feedback and see if we can get next steps. > > > > > > Could someone please review the code and provide feedback? > > > > > > Best regards, > > > Kristian Rickert > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
