ChrisHegarty opened a new pull request, #16647: URL: https://github.com/apache/lucene/pull/16647
I need to count how many vectors a field has in a segment without opening the vector values. Today the only way to do that is` getFloatVectorValues` or `getByteVectorValues` and then call `size()`. That opens the full vector values object. For sparse fields that builds a `IndexedDISI` and can prefetch index data from disk. Elasticsearch hits this on cluster stats: it walks every dense vector field on every segment just to sum counts, and on remote-backed storage each open can pull a whole cache region. We worked around it by skipping counts on stateless and caching on stateful, but the count is already in the segment metadata next to `vectorDataLength`, which `getOffHeapByteSize` reads for free. This change adds `KnnVectorsReader.getVectorCount(FieldInfo)`. Standard formats override it to return the size already loaded from the metadata at segment open. No index format change. The default implementation still opens vector values as a fallback so custom codecs stay correct.. I followed the same pattern as `getOffHeapByteSize`: one method on `KnnVectorsReader`, with overrides in Lucene99, Lucene104, backward codecs, SimpleText, Dedup, Faiss, and the usual wrappers. I added `testVectorCount` in `BaseKnnVectorsFormatTestCase`. It indexes dense fields and sparse fields in the same segment and checks `getVectorCount` matches opening the values. `getNumVectors` in the test base now uses the new API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
