Re: Force content decoding

[Timothe Litt via curl-library]

It sounds like you're on the wrong track.










Are you specifying the correct accept-encoding header on the download?






Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

On 07-Jul-22 05:19, Wolf Vollprecht wrote:

I'm not familiar with that service.  I would expect it to return what you provide. If you upload gzip'd data, the server should return gzip'd data - it's a blob.  This makes sense if you want to minimize stored size on the server. If you want to upload an uncompressed binary blob and get back the uncompressed data, but use gzip on the wire - consider transfer-encoding.  This will store the uncompressed data on the server - which may be what you're trying to avoid. Did you use the correct content-encoding (or transfer-encoding) headers when you uploaded the blobs? As for changing incoming headers - I don't think that's doable, but in any case, it's a bad idea.  "There be dragons there." curl uses zlib for encoding/decoding, so aside from convenience, you would get nothing from forcing a path thru the internal functions.

Hi Timothe,





Best,

Wolf

Thanks for your message. Indeed, I want to decode the data. The server is not broken (it’s the Github OCI registry server), I just cannot control what headers it sends for the binary blobs that I uploaded. Ideally I would have wanted to re-use the internal libcurl functions for decoding that would be used if the server would reply with the correct Content-Encoding vs implementing it in my WRITEFUNCTION.

On 7. Jul 2022, at 11:15, Timothe Litt via curl-library <curl-library@lists.haxx.se> wrote:

On 07-Jul-22 03:56, Wolf Vollprecht via curl-library wrote:

Hi all,

I have some data that I am downloading where I know that it is gzip-encoded. 
However, the server does not respond with the correct header (and I don’t have 
control over it). I was wondering if there is a way to either “manually” append 
the proper Content-Encoding header to the response or force libcurl to encode 
the response with gzip?

Best,

Wolf

For the command line:

    curl https://foo.example.com/bar | gunzip > data




Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html

I assume you mean "decode", not "encode". After the mandatory lecture about not tolerating broken servers... If you are using libcurl, depending on your requirements see zlib (libz) with CURLOPT_WRITEFUNCTION or popen() gunzip (or gzip -d).

OpenPGP_signature
Description: OpenPGP digital signature

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html