thinkharderdev commented on issue #6934: URL: https://github.com/apache/arrow-rs/issues/6934#issuecomment-2569261992
> Could you expand on this a bit, the multi-part upload is typically more performant for large uploads (>10MB). Not only can it upload chunks in parallel, it has much better reliability as it can retry just failed parts instead of the entire upload. My understanding (which may be wrong) is that read latency can be higher when doing range fetches for multi-part files that cross part boundaries since S3 stores the parts separately. On a separate note, I think there are some concurrency issues in `BufWriter` which I am investigating. We have seen issues where the parts are in the wrong order when uploaded (and it stops happening when we set the max concurrency in `BufWriter` to `1`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
