james-rms opened a new issue, #504: URL: https://github.com/apache/arrow-rs-object-store/issues/504
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** When performing a streaming upload to GCS, the best option now is to use [Multipart Uploads](https://cloud.google.com/storage/docs/multipart-uploads). This works the same way it does for S3, and has the same limitations: you cannot perform an upload with >10000 parts. This presents a challenge for large uploads with unknown size (which I happen to work with). You have to pick a part size large enough to allow the largest plausible upload size, but still small enough that your buffers don't grow too large, and your cost of retrying an part upload is not too great. The [Resumable upload] API does not have this limitation, you can upload as many parts as you want at whatever size you want. The downside of resumable uploads is that part uploads must be done serially, you cannot upload parts in parallel or out-of-order. **Describe the solution you'd like** <!-- A clear and concise description of what you want to happen. --> I'd like the option to use resumable uploads when using `put_multipart`. I would configure this in the client builder. **Describe alternatives you've considered** <!-- A clear and concise description of any alternative solutions or features you've considered. --> **Additional context** <!-- Add any other context or screenshots about the feature request here. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
