xande commented on PR #16915: URL: https://github.com/apache/lucene/pull/16915#issuecomment-6502520170
Hey Tony, Thanks for posting, this is quite promising! I did my own tests and reproduced the results on the same hardware class. I also tested the PR on an internal dataset with 4096 dimensions. **Setup.** m6gd.8xlarge (Graviton2, 32 cores, 32 MB L3), Corretto 25, this PR at `8d19cdd0`, luceneutil `KnnGraphTester`. 1M docs, M=32, beamWidth 100, `PipnnParams.DEFAULT` with alpha 1.13. A normal `forceMerge` reuses the largest segment graph. Thus, to time a full HNSW build, I merged with `addIndexes` and hid the source graphs from the merger. **Cohere v3 1024d.** The results reproduce. - 16 workers: HNSW 169.6 s, PiPNN 87.3 s, so 1.94× (you report 1.96×). - The speedup increases from 1.64× at 1 worker to 2.61× at 32 workers. - Recall@100 and nodes visited agree with your table to ±0.001 and ±10. The level-0 degree is 42.1 against 39.5. - HNSW on the merge-base commit is within 1% of HNSW on this PR. **Internal 4096-dim dataset** (1M docs, 6K real queries). | Workers | HNSW | PiPNN 10:3 | PiPNN 6:2 | |---|---|---|---| | 16 | 188.8 s | 278.2 s (0.68×) | 131.6 s (1.43×) | | 32 | 156.5 s | 158.3 s (0.99×) | 80.1 s (1.95×) | - As on your VDBBench 4096d set, HNSW makes a sparse graph here (level-0 degree 18.2). With 10:3, PiPNN is slower to build. - But the PiPNN graph is better. At about the same latency (2.1 ms, fanout 0), recall@100 is 0.900 against 0.832 for HNSW. At fanout 200, it is 0.967 against 0.924. - 6:2 is also better than HNSW on the recall/latency curve: 0.882 at 1.99 ms against 0.832 at 2.06 ms. For high dimensions, 6:2 looks like a good default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
