flyrain commented on issue #4346: URL: https://github.com/apache/iceberg/issues/4346#issuecomment-1070508331
Not sure if we've already used crc file. I'd think it can increase the comparison reliability. Here is the reason: 1. We can NOT tell two files are identical if their crc values are the same due to the high collision rate of crc. But we can tell two files are different if their crc are different, which is still useful. For example, we've known the crc values of valid file a, b, c under a dir. We then have the confidence to delete any files without crc or without a valid crc value. 2. Each file generated by Spark has its crc file, so we don't have to calculate them on the fly. We can even preload them in manifest file if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
