kdn36 commented on PR #633: URL: https://github.com/apache/arrow-rs-object-store/pull/633#issuecomment-3873523812
On performance: (1) On a local dev machine (minio same-host, backed by memory tmpfs, so no IO latency), the time to sink 2.3 GB drops from ~1.7s (SHA256) to ~1.3s (CRC64NVME). (2) On AWS, using a high-end instance (100 Gbps NIC), on standard S3, we are pushing almost 2 GB/s write for a 11.2 GB file. Results (in seconds, sorted by median) ``` Checksum Type | Best | Median | Worst --------------|---------|--------|------- unsigned | 5.22 | 7.14 | 8.41 crc64nvme | 4.41 | 7.32 | 8.95 sha256 | 5.35 | 7.83 | 9.57 default | 5.54 | 8.20 | 10.51 ``` (where defaults = signed + sha256 checksum, unsigned = not signed no checksum, sha256 = not signed but sha256 checksum, and crc64nvme = not signed but crc64nvme checksum). Note that the results on AWS have high variance, which is not unexpected given that this is in a shared production environment. The results are less pronounced because of IO, but keep in mind we keep optimizing the IO stack. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
