jimczi opened a new pull request, #16769: URL: https://github.com/apache/lucene/pull/16769
#16749 made `Lucene99FlatVectorsReader.getMergeInstance()` map the raw vectors again for a merge, so a merge no longer changes the advice on the mapping searches read at random. Two kinds of readers never got there. Readers that wrap a raw reader returned `this` from `getMergeInstance()`, so the raw reader's merge instance was never asked for. `Lucene104ScalarQuantizedVectorsReader` and `FaissKnnVectorsReader` now hand merges a reader over the raw reader's merge instance and pass `finishMerge()` on. A merge reads the raw vectors in both: to quantize them again, and to rebuild the Faiss index. The Faiss index file itself is not read by a merge and is left alone. The de-duplicating formats read their own data files. `DedupFlatVectorsReader` and `DedupScalarQuantizedVectorsReader` now open them again for merges, through a small `MergeInput` both share. That mapping only drops the random advice. It does not claim the merge reads front to back and once, because a merge in this format comes back to vectors it has already read, to compare the ones whose hashes collide and to quantize the distinct ones at the end. A merge now maps a file again only when searches read it at random, which is where the conflict comes from. `Lucene99FlatVectorsReader` checks this too, so a flat format used without a graph, or a reader a merge opened itself, keeps reading through the mapping it has. The backward-codecs readers do not ask for random reads and are unchanged. `TestVectorsMergeReadAdvice` now covers the quantized format and the two cases that must not map again. `TestDedupVectorsMergeReadAdvice` covers both de-duplicating formats, with vectors that repeat across segments, and checks the merged vectors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
