jordanisaacs commented on issue #281: URL: https://github.com/apache/arrow-rs-object-store/issues/281#issuecomment-5379008547
I'll add support for this feature here. There are use cases for streaming puts for out of memory object store writes. It has two benefits: 1) Improved latency - Ability to upload while reading/streaming data 2) Reduced memory usage - Ability to discard data as it is uploaded Now as stated by @/senzzzi you can use multipart uploads but there is a real cost concern there. And there is the fact that conditional writes on multipart isn't supported on all object stores (see GCP). And what if I don't to provide 1MB of memory upfront for a part. There is the drawback of losing transparent retries if you are optimizing for (2), but that is a tradeoff that should be supported without having to reach for the hammer that is multipart upload. And there is such thing as [resumable uploads](https://docs.cloud.google.com/storage/docs/resumable-uploads) on GCP which does support retrying. I can see a route here where there is streaming APIs with optional retries - either store supported resumable, or a way to buffer the stream if you don't care about (2). Even supporting streaming on multipart doesn't sound like a bad idea. Now you can get the full matrix of streaming vs. boxed, resumable vs. not resumable, retryable vs. not retryable, multipart vs. single put, and conditional write support. While not all combinations are supported by all object stores, it is a unified matrix that covers every use case in a single api. If S3 came out tomorrow with resumable uploads or GCP with condiitonal writes on multipart uploads, you could still have a cloud agnostic api. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
