On 30.07.21 03:30, 積丹尼 Dan Jacobson wrote:
The man page should mention for --compression=gzip, the website can still ignore it and send the whole file uncompressed. And wget is not picky, and still will accept it. So better have extra disk space ready. Tested with https://www.ncc.gov.tw/chinese/files/opendata/radio.csv .
This is because the client just tells the server what it supports / accepts (not what it desperately wants).
The issue here is that the "identity" encoding is always implicit - there is no way to disable it.
So a Accept-Encoding: gzipmeans "hey server, send me the content either not encoded or gzip encoded - whatever you like :-)"
If you know that the server sends uncompressed content, you can compress it yourself on-the-fly to avoid excessive disk space usage. At least it works for single files:
wget -O- <URL> | gzip > radio.csv.gzPrinting a warning may help some users, but others will be swamped, e.g. when doing recursive downloads.
Regards, Tim
OpenPGP_signature
Description: OpenPGP digital signature
