thinkharderdev opened a new issue, #281:
URL: https://github.com/apache/arrow-rs-object-store/issues/281
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always
frustrated when [...]
(This section helps Arrow developers understand the context and *why* for
this feature, in addition to the *what*)
-->
Currently, `put/put_opts` takes a `PutPayload` which is just a wrapper
around an array of `Bytes`. It would be useful (for us) to be able to stream
the body without having to do a multipart upload.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
The underlying http library `reqwest` already supports this. So I think we
can change the definition of `PutPayload` to an enum like so:
```
type StreamingBody = Box<dyn futures::stream::Stream<Item = Result<Bytes,
Box<dyn std::error::Error + Send + Sync + 'static>>> + Send + 'static>;
pub enum PutPayload {
Bytes(Arc<[Bytes]>),
Streaming(StreamingBody,usize),
}
```
where `PutPayload::Streaming` holds the stream and the total content length
(which must be known ahead of time).
The (unfortunate) requirement of the S3 API to know the `content-length` up
front means this isn't fully general but can be useful in certain situations
such as buffering a largeish file on disk before uploading. In that case, a
streaming body would allow the upload to not have to load the entire object in
memory to upload it.
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features
you've considered.
-->
This can already be done with `put_multipart` but mutli-part upload is
complicated, may have some performance implications for reads.
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
