mikemccand commented on PR #15979: URL: https://github.com/apache/lucene/pull/15979#issuecomment-5061802762
> > I'm really curious if with this new flat vectors format we discover Cohere has some dup vectors! > > I indexed the first 1M Cohere v3 vectors (1024 dimensions) -- and punched through to unique vector count (getFloatVectorValues -> getGroupView -> size), which reported 999987 unique vectors (i.e. 13 / 1M are duplicates). > > Also verified this using a simple numpy script generated by Claude, which reports the same unique vector count. Whoa! Cool, thanks for checking. Now I want to know which 13 wiki articles those were :) Do you have their titles maybe? The luceneutil Cohere v3 processed corpus files has a metadata file w/ title and body and such corresponding to each vector. Was it all 13 are identical, or, somehow pair wise and one triple-wise... or some other fascinating grouping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
