kaivalnp commented on PR #15979: URL: https://github.com/apache/lucene/pull/15979#issuecomment-5047918716
> Will `CheckIndex` print summary stats like number of vectors and number of post-dedup (unique) vectors? It does not do so today -- it will only report the [total number of vectors](https://github.com/apache/lucene/blob/224f580a5d59fa2747c3da98dc1c6461c067a9b3/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java#L3126-L3136). This might be a good addition though! > I'm really curious if with this new flat vectors format we discover Cohere has some dup vectors! I indexed the first 1M Cohere v3 vectors (1024 dimensions) -- and punched through to unique vector count (`getFloatVectorValues -> getGroupView -> size`), which reported `999987` unique vectors (i.e. 13 / 1M are duplicates). Also verified this using a simple `numpy` script generated by Claude, which reports the same unique vector count. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
