ByteBaker commented on issue #783:
URL:
https://github.com/apache/arrow-rs-object-store/issues/783#issuecomment-4896493679
Upon looking further, it turns out the surface area is far bigger than
anticipated. `reqwest` defines the methods
`ClientBuilder::no_{gzip|brotli|zstd|deflate()` like the following.
```rust
pub fn no_gzip(self) -> ClientBuilder {
#[cfg(feature = "gzip")]
{
self.gzip(false)
}
#[cfg(not(feature = "gzip"))]
{
self
}
}
```
This means that the said feature must be enabled for reqwest to process the
given encoding.
However, `object_store` does not yet provide a way to opt into those
reqwest-specific features when using `object_store` as a dependency. To ensure
that a user externally enabling those features in reqwest via a parallel
dependency does not change reqwest's behavior, it chooses to opt-out of all
those algorithms. This is declared in the comment preceding the code:
```rust
// Explicitly disable compression, since it may be automatically enabled
// when certain reqwest features are enabled. Compression interferes
// with the `Content-Length` header, which is used to determine the
// size of objects.
builder = builder.no_gzip().no_brotli().no_zstd().no_deflate();
```
Having said that, this is clear that object-store intentionally disables all
forms of response compression. There's 3 ways forward to this:
1. Explicitly declare in client builders' docs that the user must avoid
adding an `Accept-Encoding` header since object store doesn't support it.
@nkemnitz's concern is valid, but the contract between the headers and
compression features is not enforced by them. (**Easiest, but errors can happen
if users don't read it**)
2. Remove `Accept-Encoding` header while storing default headers for the
client and declare the behaviour in docs. This still stays with no compression,
but the users know that including this header in default headers has no effect,
and the client still requests content w/o compression. (**Easy, ensures errors
don't happen**)
3. Object store adds feature flags that optionally enable the said features
in reqwest. Along with exploring and handling the `Content-Length` problem
mentioned in the comment. (**Complex, complete solution to the problem**)
Please suggest a way forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
