dengfengjiang77 commented on issue #16674: URL: https://github.com/apache/lucene/issues/16674#issuecomment-6020835337
> I think there is another full read from `checkIntegrity`? A step 0, run (sequentially?) on all the input segments before they are merged? That is a scan of the whole file and then confirming the checksum? But it's nearly redundant because any of the existing steps 1, 2, 3 could validate the checksum after their full scan? But ... it'd be a change in error detection timing because it would have done some merge work before detecting the bit flips. Thanks, Mike, I think replace the full read in step 0 is worth trying, but step1 and step 3 only read live vectors, CRC need every bytes in order. for these files we need be careful: .vec file - move the check from step 0 to step1, the raw copy can sequentially read , update CRC32 .veq file - we don't read in the merge, it is recomputed by .vec, check it in step 0 only for early detection. Could we drop this check? .vex file - it will used to initialize the HNSW, suggest we keep the full read check for it I'm measuring how much of step 0 is .vec vs .veq and will post the numbers later . If this direction looks good to you, I can open a separate issue for it and keep this PR to the centroid change(remove step2 only). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
