iprithv commented on code in PR #16065:
URL: https://github.com/apache/lucene/pull/16065#discussion_r3308835446
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissLibraryNativeImpl.java:
##########
@@ -275,31 +268,34 @@ private interface FloatToFloatFunction {
private final Arena arena;
private final MemorySegment indexPointer;
private final FloatToFloatFunction scaler;
+ private final VectorSimilarityFunction function;
+ private final int dimension;
private boolean closed;
- private Index(MemorySegment indexPointer) {
+ private Index(MemorySegment indexPointer, VectorSimilarityFunction
function) {
this.arena = Arena.ofShared();
this.indexPointer =
indexPointer
// Ensure timely cleanup
.reinterpret(arena, wrapper::faiss_Index_free);
-
- // Get underlying function
- int metricType = wrapper.faiss_Index_metric_type(indexPointer);
- VectorSimilarityFunction function = metricToFunction(metricType);
Review Comment:
@kaivalnp, benchmark (50k docs, 384 dims)
The original ~10k doc run was indeed too small to show meaningful HNSW
advantages. At 50k docs:
| Format | Metric | Latency | QPS | Recall@10 |
|--------|--------|---------|-----|-----------|
| Faiss-Flat | DOT_PRODUCT | 7.78 ms | 129 | 1.000 |
| Faiss-HNSW | DOT_PRODUCT | 0.14 ms | 7,069 | 0.229 |
| Lucene-HNSW | DOT_PRODUCT | 0.11 ms | 9,132 | 0.199 |
| Faiss-Flat | COSINE | 7.74 ms | 129 | 1.000 |
| Faiss-HNSW | COSINE | 0.15 ms | 6,817 | 0.210 |
| Lucene-HNSW | COSINE | 0.08 ms | 13,286 | 0.179 |
HNSW is **55-100x faster** than flat at this scale.
**2. Lucene vs Faiss HNSW** (both with M=16, efConstruction=128)
- Faiss indexes ~2.5x faster (native C++/OpenMP)
- Lucene searches slightly faster in this setup
- Recall is comparable (~0.2 for both with defaults)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
