Hi,> > curl.htm is the desired content, whereas wget.htm contains a generic 404
> error page "Page cannot be found", suggesting wget is not redirecting to > the correct URL.
Wget2 received couple of fixes in the past weeks. With the current code (not yet released), wget2 downloads the page correctly.
The README.md contains recipies to build Wget2 with the help of docker. No need to install dev tools. Regards, Tim On 8/23/26 13:00, [email protected] wrote:
Hi,Thanks for looking into this, it seems though that wget fails to use the correct URL anyway, as when using "--content-on-error=on", a generic 404 error page is returned by the server instead of the blog post.Here is an example using a better URL without the encoding. This produces curl.htm and wget.htm:curl -L 'https://www.bbc.co.uk/blogs/adamcurtis/entries/ a2094c9d-9864-348e-a241-7aa93adf0c09 <https://www.bbc.co.uk/blogs/ adamcurtis/entries/a2094c9d-9864-348e-a241-7aa93adf0c09>' > curl.htmwget --content-on-error=on 'https://www.bbc.co.uk/blogs/adamcurtis/ entries/a2094c9d-9864-348e-a241-7aa93adf0c09 <https://www.bbc.co.uk/ blogs/adamcurtis/entries/a2094c9d-9864-348e-a241-7aa93adf0c09>' -O wget.htmcurl.htm is the desired content, whereas wget.htm contains a generic 404 error page "Page cannot be found", suggesting wget is not redirecting to the correct URL.Thanks On Sun, Aug 23, 2026, at 11:45 AM, Tim Rühsen wrote:Hi, > Wget fails to download pages from Adam Curtis' archived blog, such as this: This is expected since the server answers with 404 instead of 200 and wget dicards any content in this case (by default). To do store the content on error, you have to add `--content-on-error` on the command line. Cheers, TimOn 8/22/26 13:24, [email protected] <mailto:[email protected]> wrote: > Wget fails to download pages from Adam Curtis' archived blog, such as this:>> https://www.bbc.co.uk/webarchive/ https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>> > But Curl succeeds. This may be because it is re-encoding the % symbols. > > output of wget -d: > >> 22.122345.635 Local URI encoding = 'UTF-8' >> 22.122345.635 Input URI encoding = 'UTF-8'>> 22.122345.635 Fetched HSTS data from '/home/miles/.local/share/ wget/.wget-hsts' >> 22.122345.635 Fetched HPKP data from '/home/miles/.local/share/ wget/.wget-hpkp'>> 22.122345.635 set_exit_status(0)>> 22.122345.635 path webarchive/ https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9 -> >> 22.122345.635 webarchive/ https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9 >> 22.122345.635 local filename = 'https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9' >> 22.122345.635 host_add_job: job fname https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9 >> 22.122345.635 host_add_job: 0x5622fb64ec70 https://www.bbc.co.uk/ webarchive/ https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>>> 22.122345.635 host_add_job: qsize 1 host-qsize=1 >> 22.122345.635 queue_size: qsize=1 >> 22.122345.635 queue_size: qsize=1 >> 22.122345.635 queue_size: qsize=1 >> 22.122345.635 [0] action=1 pending=0 host=0x0 goaway=n/a>> 22.122345.635 dequeue job https://www.bbc.co.uk/webarchive/ https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>>> 22.122345.635 resolving www.bbc.co.uk:443 <http://www.bbc.co.uk:443>... >> 22.122345.643 has 212.58.237.1:443 >> 22.122345.643 has 212.58.236.129:443 >> 22.122345.643 has 212.58.237.129:443 >> 22.122345.643 has 212.58.235.1:443 >> 22.122345.643 has 212.58.235.129:443 >> 22.122345.643 has 212.58.236.1:443 >> 22.122345.643 has 212.58.237.129:443 >> 22.122345.643 has 212.58.236.1:443 >> 22.122345.643 has 212.58.235.1:443 >> 22.122345.643 has 212.58.237.1:443 >> 22.122345.643 has 212.58.235.129:443 >> 22.122345.643 has 212.58.236.129:443 >> 22.122345.643 trying 212.58.237.1:443... >> 22.122345.659 GnuTLS init >> 22.122345.835 GnuTLS system certificate store is empty >> 22.122345.835 Certificates loaded: 388 >> 22.122345.835 GnuTLS init done >> 22.122345.836 TLS False Start requested >> 22.122345.836 SNI www.bbc.co.uk <http://www.bbc.co.uk>>> 22.122345.836 OCSP stapling requested for www.bbc.co.uk <http:// www.bbc.co.uk>>> 22.122345.836 ALPN offering h2 >> 22.122345.836 ALPN offering http/1.1 >> 22.122345.894 OCSP stapling is not supported by 'www.bbc.co.uk' >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 host has no pubkey pinnings stored in hpkp db >> 22.122345.894 TLS False Start: off >> 22.122345.894 ALPN: Server accepted protocol 'h2' >> 22.122345.895 Handshake completed>> 22.122345.895 established connection www.bbc.co.uk <http:// www.bbc.co.uk> >> 22.122345.895 cookie_create_request_header for host=www.bbc.co.uk <http://www.bbc.co.uk> path=webarchive/ https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9>> 22.122345.895 HTTP2 stream id 1 >> 22.122345.895 [0] action=1 pending=1 host=0x5622fb64e6b0 goaway=false >> 22.122345.895 [0] action=2 pending=1 host=0x5622fb64e6b0 goaway=false >> 22.122345.895 ## pending_requests = 1 >> 22.122345.895 [FRAME 0] > SETTINGS >> 22.122345.895 [FRAME 0] > WINDOW_UPDATE >> 22.122345.895 [FRAME 1] > HEADERS >> 22.122345.895 [FRAME 1] > :method: GET>> 22.122345.895 [FRAME 1] > :path: /webarchive/ https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>> 22.122345.895 [FRAME 1] > :scheme: https>> 22.122345.895 [FRAME 1] > :authority: www.bbc.co.uk <http:// www.bbc.co.uk>>> 22.122345.895 [FRAME 1] > accept-encoding: gzip, deflate, br, zstd >> 22.122345.895 [FRAME 1] > accept: */* >> 22.122345.895 [FRAME 1] > user-agent: Wget/2.2.1 >> 22.122345.909 [FRAME 0] < SETTINGS >> 22.122345.909 [FRAME 0] < WINDOW_UPDATE >> 22.122345.909 [FRAME 0] > SETTINGS >> 22.122345.910 [FRAME 0] < SETTINGS >> 22.122346.113 :status: 404 >> 22.122346.113 date: Sat, 22 Aug 2026 11:23:46 GMT >> 22.122346.113 content-type: text/html;charset=utf-8 >> 22.122346.113 content-length: 9687 >> 22.122346.113 belfrage-cache-status: MISS >> 22.122346.113 bid: bruce >> 22.122346.113 brequestid: 268b641cf47a4cd78df3312be105da1c >> 22.122346.113 bsig: b24bbece8fc068ce8b9dfd6803e8e57e>> 22.122346.113 cache-control: public, stale-if-error=90, stale- while-revalidate=30, max-age=5>> 22.122346.113 content-encoding: gzip >> 22.122346.113 req-svc-chain: GTM,BELFRAGE,MOZART >> 22.122346.113 vary: accept-encoding >> 22.122346.114 via: 2.0 BBC-GTM, 1.1 Belfrage >> 22.122346.114 x-content-type-options: nosniff >> 22.122346.114 x-mrid: n1 >> 22.122346.114 x-xss-protection: 1; mode=block >> 22.122346.114 origin-agent-cluster: ?0 >> 22.122346.114 x-lb-nocache: true>> 22.122346.114 nel: {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":0.25}>> 22.122346.114 x-bbc-edge-cache-status: MISS >> 22.122346.114 x-bbc-origin-response-status: 404>> 22.122346.114 report-to: {"group":"default","max_age":2592000,"endpoints":[{"url":"https:// default.bbc-reporting-api.app/report-endpoint <https://default.bbc- reporting-api.app/report- endpoint>","priority":1}],"include_subdomains":true <https:// default.bbc-reporting-api.app/report-endpoint%22,%22priority%22:1%7D <https://default.bbc-reporting-api.app/report-endpoint%22, %22priority%22:1%7D>],%22include_subdomains%22:true>}>> 22.122346.114 server: BBC-GTM >> 22.122346.114 strict-transport-security: max-age=31536000; preload>> 22.122346.114 permissions-policy: browsing-topics=(), join-ad- interest-group=(), run-ad-auction=()>> 22.122346.114 x-robots-tag: bingbot: noarchive >> 22.122346.114 [FRAME 1] < HEADERS >> 22.122346.114 closing stream 1 >> 22.122346.114 ## response status 404 >> 22.122346.114 set_exit_status(8) >> 22.122346.114 keep_alive=1>> 22.122346.114 update HSTS www.bbc.co.uk:443 <http:// www.bbc.co.uk:443> (maxage=31536000, includeSubDomains=0)>> 22.122346.114 _host_remove_job: 0x5622fb64ec70 >> 22.122346.114 host_remove_job: qsize=0 host->qsize=0 >> 22.122346.114 [0] action=1 pending=0 host=0x5622fb64e6b0 goaway=false >> 22.122346.114 closing connection >> 22.122346.115 [0] action=1 pending=0 host=0x0 goaway=n/a >> 22.122346.115 main: wake up >> 22.122346.115 main: done>> 22.122346.115 Successfully updated '/home/miles/.local/share/ wget/.wget-hsts'. >> 22.122346.115 Saved 1 HSTS entry into '/home/miles/.local/share/ wget/.wget-hsts' >> 22.122346.115 blacklist https://www.bbc.co.uk/webarchive/ https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>>> ---- >> Certificate info [0]: >> Valid since: Mon 13 Jul 2026 14:21:36 BST >> Expires: Thu 28 Jan 2027 13:21:36 GMT >> Fingerprint: e3756b8bc1f345cc3018a2b146b88ca5 >> Serial number: 518996aacb43bd1f78166375 >> Public key: EC/ECDSA, High (256 bits) >> Version: #3>> DN: C=GB,ST=London,L=London,O=BRITISH BROADCASTING CORPORATION,CN=www.bbc.co.uk <http://www.bbc.co.uk> >> Issuer's DN: C=BE,O=GlobalSign nv-sa,CN=GlobalSign ECC OV SSL CA 2018>> Issuer's OID: 2.5.4.6 >> Issuer's UID: 2.5.4.6 >> Certificate info [1]: >> Valid since: Wed 21 Nov 2018 00:00:00 GMT >> Expires: Tue 21 Nov 2028 00:00:00 GMT >> Fingerprint: 9635de986bc7f9b19f09aec50e3a3909 >> Public key: EC/ECDSA, Ultra (384 bits) >> Version: #3 >> DN: C=BE,O=GlobalSign nv-sa,CN=GlobalSign ECC OV SSL CA 2018>> Issuer's DN: OU=GlobalSign ECC Root CA - R5,O=GlobalSign,CN=GlobalSign>> Issuer's OID: 2.5.4.11 >> Issuer's UID: 2.5.4.11 >> Certificate info [2]: >> Valid since: Wed 21 Nov 2018 00:00:00 GMT >> Expires: Sun 18 Mar 2029 10:00:00 GMT >> Fingerprint: 8f665697f9d7f2c6c6b96a99259b7ffc >> Serial number: 01ee5f223e3d355b992bc82ab4 >> Public key: EC/ECDSA, Ultra (384 bits) >> Version: #3 >> DN: OU=GlobalSign ECC Root CA - R5,O=GlobalSign,CN=GlobalSign >> Issuer's DN: OU=GlobalSign Root CA - R3,O=GlobalSign,CN=GlobalSign >> Issuer's OID: 2.5.4.11 >> Issuer's UID: 2.5.4.11 >> Certificate info [3]: >> Valid since: Wed 19 Sep 2018 01:00:00 BST >> Expires: Fri 28 Jan 2028 12:00:00 GMT >> Fingerprint: 58bfb0eacf17e2d0a65a4d061042c3bf >> Serial number: 01ee5f169dff97352b6465d66a >> Public key: RSA, Medium (2048 bits) >> Version: #3 >> DN: OU=GlobalSign Root CA - R3,O=GlobalSign,CN=GlobalSign>> Issuer's DN: C=BE,O=GlobalSign nv-sa,OU=Root CA,CN=GlobalSign Root CA>> Issuer's OID: 2.5.4.6 >> Issuer's UID: 2.5.4.6 >> ---- >> Ephemeral ECDH using curve (null) >> Key Exchange: ECDHE-RSA >> Protocol: TLS1.3 >> Certificate Type: X.509 >> Cipher: AES-256-GCM >> MAC: AEAD >> ---->> [0] Downloading 'https://www.bbc.co.uk/webarchive/ https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>' ... >> HTTP ERROR response 404 [https://www.bbc.co.uk/webarchive/ https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>]>> *Attachments:* * OpenPGP_signature.asc
OpenPGP_signature.asc
Description: OpenPGP digital signature
