To my knowledge, FAISS isn't utilizing hand-rolled SIMD calculations. Do we know if it was compiled with `--ffast-math`?
Vespa does utilize SIMD optimizations for vector comparisons. Some more ways I think Lucene is slower (though, I am not sure the 2x is fully explained): - Reading floats onto heap float[] instead of accessing Memory Segments directly when scoring - We store the graph in a unique way that requires a decoding step when exploring a new candidate, reading in vints and doing a binary search. I think all other hnsw impls do flat arrays of int/long values. - We always use SparseBitSet, which for smaller indices <1M can have a noticeable impact on performance. I have seen this in my own benchmarking (switching to fixedbitset measurably improved query times on smaller data sets) Both of these are fairly "cheap". Which might explain the FAISS 10% difference. However, I am not sure they fully explain the 2x difference with vespa. On Thu, Jun 19, 2025 at 3:37 PM Adrien Grand <jpou...@gmail.com> wrote: > Thanks Mike, this is useful information. Then I'll try to reproduce this > benchmark to better understand what is happening. > > On Thu, Jun 19, 2025 at 8:16 PM Michael Sokolov <msoko...@gmail.com> > wrote: > >> We've recently been comparing Lucene's HNSW w/FAISS' and there is not >> a 2x difference there. FAISS does seem to be around 10-15% faster I >> think? The 2x difference is roughly what I was seeing in comparisons >> w/hnswlib prior to the dot-product improvements we made in Lucene. >> >> On Thu, Jun 19, 2025 at 2:12 PM Adrien Grand <jpou...@gmail.com> wrote: >> > >> > Chris, >> > >> > FWIW I was looking at luceneknn ( >> https://github.com/erikbern/ann-benchmarks/blob/f402b2cc17b980d7cd45241ab5a7a4cc0f965e55/ann_benchmarks/algorithms/luceneknn/Dockerfile#L15) >> which is on 9.7, though I don't know if it enabled the incubating vector >> API at runtime? >> > >> > I hope that mentioning ANN benchmarks did not add noise to this thread, >> I was mostly looking at whether I could find another benchmark that >> suggests that Lucene is significantly slower in similar conditions. Does it >> align with other people's experience that Lucene is 2x slower or more >> compared with other good HNSW implementations? >> > >> > Adrien >> > >> > Le jeu. 19 juin 2025, 18:44, Chris Hegarty >> <christopher.hega...@elastic.co.invalid> a écrit : >> >> >> >> Hi Adrien, >> >> >> >> > Even though it uses Elasticsearch to run the benchmark, it really >> benchmarks Lucene functionality, >> >> >> >> Agreed. >> >> >> >> > This seems consistent with results from >> https://ann-benchmarks.com/index.html though I don't know if the cause >> of the performance difference is the same or not. >> >> >> >> On ann-benchmarks specifically. Unless I’m looking in the wrong place, >> then they’re using Elasticsearch 8.7.0 [1], which predates our usage of the >> Panama Vector API for vector search. We added support for that in Lucene >> 9.7.0 -> Elasticsearch 8.9.0. So those benchmarks are wildly out of date, >> no ? >> >> >> >> -Chris. >> >> >> >> [1] >> https://github.com/erikbern/ann-benchmarks/blob/f402b2cc17b980d7cd45241ab5a7a4cc0f965e55/ann_benchmarks/algorithms/elasticsearch/Dockerfile#L2 >> >> >> >> >> >> > On 19 Jun 2025, at 16:39, Adrien Grand <jpou...@gmail.com> wrote: >> >> > >> >> > Hello all, >> >> > >> >> > I have been looking at this benchmark against Vespa recently: >> https://blog.vespa.ai/elasticsearch-vs-vespa-performance-comparison/. >> (The report is behind an annoying email wall, but I'm copying relevant data >> below, so hopefully you don't need to download the report.) Even though it >> uses Elasticsearch to run the benchmark, it really benchmarks Lucene >> functionality, I don't believe that Elasticsearch does anything that >> meaningfully alters the results that you would get if you were to run >> Lucene directly. >> >> > >> >> > The benchmark seems designed to highlight the benefits of Vespa's >> realtime design, that's fair game I guess. But it also runs some queries in >> read-only scenarios when I was expecting Lucene to perform better. >> >> > >> >> > One thing that got me curious is that it reports about 2x worse >> latency and throughput for pure unfiltered vector search on a force-merged >> index (so single segment/graph). Does anybody know why Lucene's HNSW may >> perform slower than Vespa's HNSW? This seems consistent with results from >> https://ann-benchmarks.com/index.html though I don't know if the cause >> of the performance difference is the same or not. >> >> > >> >> > For reference, here are details that apply to both Lucene and >> Vespa's vector search: >> >> > - HNSW, >> >> > - float32 vectors, no quantization, >> >> > - embeddings generated using Snowflake's Arctic-embed-xs model >> >> > - 1M docs >> >> > - 384 dimensions, >> >> > - dot product, >> >> > - m = 16, >> >> > - max connections = 200, >> >> > - search for top 10 hits, >> >> > - no filter, >> >> > - single client, so no search concurrency, >> >> > - purple column is force-merged, so single segment/graph like Vespa. >> >> > >> >> > I never seriously looked at Lucene's vector search performance, so >> I'm very happy to be educated if I'm making naive assumptions! >> >> > >> >> > Somewhat related, is this the reason why I'm seeing many threads >> around bringing 3rd party implementations into Lucene, including ones that >> are very similar to Lucene on paper? To speed up vector search? >> >> > >> >> > -- >> >> > Adrien >> >> > <vespa-vs-es-screenshot.png> >> >> > --------------------------------------------------------------------- >> >> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >> > For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> > > -- > Adrien >
