kdn36 opened a new issue, #611: URL: https://github.com/apache/arrow-rs-object-store/issues/611
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** We are sinking large dataframes to S3. For AWS, SHA256 is the only supported checksum. When profiling a high-throughput setup (100 Gbps NIC), the checksum adds 25-30% overhead as measured by AWS upload times, compared with not using a checksum. **Describe the solution you'd like** Support `CRC-64/NVME (CRC64NVME)` (which is the AWS default). **Describe alternatives you've considered** Turning off checksum completely (`"AWS_UNSIGNED_PAYLOAD": "true"`) improves performance but adds unacceptable object integrity risk for large files using multi-part upload. **Additional context** Use case: high-end data file sink and read from AWS S3 and other S3 storage back-ends using polars dataframe library. Note. In addition, it would be nice to see SHA256 offloaded to hardware, even though not all hardware supports this. Currently all SHA256 operations are done on the CPU. This may require swapping the `ring` crate for a more modern crypto crate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
