CH-Abhinav commented on issue #16674: URL: https://github.com/apache/lucene/issues/16674#issuecomment-5751560906
do you think doing unbiased samples will always work ? maybe a 1-5% sample would be good enough to calculate centroid for OSQ but that is completely different approach and we don't know if unbiased sampling will work for all vectors we are trading accuracy which is already traded while quantizing. I would say this approach will be more optimal :- 1. if delete's are N<1% for small segments and N<0.1% for large segments we do what kaivalnp is proposing as it would be optimal to do so 2. if deleted docs are N>1% and N>0.1% for small and large segments respectively we should go with unbiased sampling with K~5%-10% of docs value taken randomly. my question is can we do this unbiased sampling initially (when first centroid is calculated from graph) and is that accuracy is good enough for lucene? may be we will need to some tests to check accuracy and performance of this method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
