john-mlika opened a new pull request, #16696: URL: https://github.com/apache/lucene/pull/16696
An HNSW merge over an asymmetric scalar-quantized field writes the merged float vectors, then reads them back and quantizes them again for the merge scorer's query side. This change produces both quantizations in one writer walk. apache/lucene#15732 stopped the merge from writing the merged vectors twice by moving query-side production from the flat writer to `QuantizedVectorsReader#getRandomVectorScorerSupplierForMerge`, introducing the read-back. `DirectIODirectory` in `lucene/misc` supports direct I/O during merges, and there the read-back is a full device read of just-written data. It is not the default, but even with buffered I/O the merged data may leave the cache before the read-back, which then goes to the device as well. At 1,000,000 vectors of dimension 128, merged `.vec` reads fall from 512,000,114 bytes to 114. Merge wall time also fell by about 2 to 3 percent, across three runs on one machine. `FlatVectorsWriter` gains a method whose default prepares nothing, so other flat writers keep the reader path. The Lucene104 writer prepares only in default `CENTERED` mode; both data-blind modes retain the reader path. The file format is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
