orlp commented on PR #633: URL: https://github.com/apache/arrow-rs-object-store/pull/633#issuecomment-3854301124
But to put some *paper napkin* numbers to this... Due to the way the code is currently architectured (this might be fundamental if the hash has to be in the header, I don't know about that), the hash must be known before the object is uploaded. This means hashing time is added to upload time. SHA256 with hardware acceleration on Intel seems to be around 2GB / sec (may vary a bit depending on machine, but let's assume this). So hashing 4GB takes 2 seconds. [`crc-fast-rust`](https://github.com/awesomized/crc-fast-rust) claims speeds of 100+ GB/s. So let's assume 100GB / s, in which case hashing 4GB takes 0.04 seconds. With a 10 Gbit / s connection uploading a 4GB object takes 3.2 seconds. So the total time taken: - SHA256: 5.2 seconds. - CRC64NVME: 3.24 seconds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
