slow-J commented on PR #16631: URL: https://github.com/apache/lucene/pull/16631#issuecomment-5587573956
Unpacking isnt used during searching IIUC, the doc is scored straight from its packed bytes without calling unpack. `PACKED_NIBBLE` is symmetric, so the query is quantized to one value per byte and left that way ([`getRandomQuantizedVectorScorer`](https://github.com/apache/lucene/blob/553fb33c0ef7c5e6d54ed4f35b966d95cb6535a9/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorScorer.java#L107-L114)) Search doesn't materialise an unpacked copy in memory, the doc stays packed. [`int4DotProductSinglePackedBody`](https://github.com/apache/lucene/blob/553fb33c0ef7c5e6d54ed4f35b966d95cb6535a9/lucene/core/src/java25/org/apache/lucene/internal/vectorization/PanamaVectorUtilSupport.java#L616-L651) loads a register of packed bytes, deriving the 2 nibbles in place, and doesn't store them. From a JFR method timing, 200K × 1024d Cohere v3, two JVM runs against the same index: | | `unpackNibbles` | `int4DotProductSinglePacked` | | --- | ---: | ---: | | index+merge+forceMerge | 91,920,896 | 3,229,056,018 | | search (2,000 q, topK 100) | 0 | 29,750,518 | I did run a search test before switching to `INDEX_ONLY`, I forget did I never post or edit it out: 500K/1024d, `quantizeBits` 4, maxConn 64, beamWidth 250, topK 100, fanout 100, 10K queries, forceMerge, `rerank=false`, 9 runs each of baseline and candidate with the index rebuilt every run. | overSample | recall base | recall cand | latency Δ | visited base | visited cand | | ---: | ---: | ---: | ---: | ---: | ---: | | 1 | 0.918 | 0.918 | −0.49% (t=−0.57) | 8,218 | 8,219 | | 2 | 0.922 | 0.922 | −1.45% (t=−1.99) | 14,145 | 14,145 | | 5 | 0.923 | 0.923 | −0.04% (t=−0.05) | 28,585 | 28,584 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
