tustvold commented on issue #6934: URL: https://github.com/apache/arrow-rs/issues/6934#issuecomment-2569220357
> This can already be done with put_multipart but mutli-part upload is complicated, may have some performance implications for reads. Could you expand on this a bit, the multi-part upload is typically more performant for large uploads. Not only can it upload chunks in parallel, it has much better reliability as it can retry just failed parts instead of the entire upload. I have debated adding this in the past, but the protocol for supporting it is [complicated](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). To be honest I had viewed the streaming upload feature as largely having been superseded by the multipart upload feature. This is certainly the way it was [marketed](https://aws.amazon.com/blogs/aws/amazon-s3-multipart-upload/) and AWS tools like [S3 Transfer Manager](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html) recommend using multipart uploads. > S3 Transfer Manager can take advantage of performance improvements such as the [multipart upload API](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) and [byte-range fetches](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
