It seems that "wget -m" uses a different "mode" to get files than the regular command.
Some sites (including sysinternals.com) now block wget requests. The example is windows, but the same holds for unix variants of wget. It seems to be the difference between the http HEAD and GET commands. Can anything be done about this? The first example is without the "-m" option and works fine, because wget is using the http GET command to retrieve the page: F:\downloads\sysinternals>wget -d -d -d -d --save-cookies cookies.txt -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" http://www.sysinternals.com/index.shtml DEBUG output created by Wget 1.9.1 on Windows. set_sleep_mode(): mode 0x80000001, rc 0x80000000 --17:50:19-- http://www.sysinternals.com/index.shtml => `index.shtml.1' Resolving www.sysinternals.com... seconds 0.00, 66.193.254.46 Caching www.sysinternals.com => 66.193.254.46 Connecting to www.sysinternals.com[66.193.254.46]:80... seconds 0.00, connected. Created socket 1916. Releasing 003926F0 (new refcount 1). ---request begin--- GET /index.shtml HTTP/1.0 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Host: www.sysinternals.com Accept: */* Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Server: Microsoft-IIS/5.0 Date: Sat, 14 Aug 2004 15:52:23 GMT X-Powered-By: ASP.NET Content-Type: text/html Length: unspecified [text/html] [ <=> ] 14,469 33.64K/s Closing fd 1916 17:50:19 (33.64 KB/s) - `index.shtml.1' saved [14469] Saving cookies to cookies.txt. Done saving cookies. The second example with the "-m" fails because wget is using the http HEAD command to retrieve the page: F:\downloads\sysinternals>wget -m -d -d -d -d --save-cookies cookies.txt -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" http://www.sysinternals.com/index.shtml DEBUG output created by Wget 1.9.1 on Windows. set_sleep_mode(): mode 0x80000001, rc 0x80000000 Enqueuing http://www.sysinternals.com/index.shtml at depth 0 Queue count 1, maxcount 1. Dequeuing http://www.sysinternals.com/index.shtml at depth 0 Queue count 0, maxcount 1. --17:56:53-- http://www.sysinternals.com/index.shtml => `www.sysinternals.com/index.shtml' Resolving www.sysinternals.com... seconds 0.00, 66.193.254.46 Caching www.sysinternals.com => 66.193.254.46 Connecting to www.sysinternals.com[66.193.254.46]:80... seconds 0.00, connected. Created socket 1916. Releasing 00392900 (new refcount 1). ---request begin--- HEAD /index.shtml HTTP/1.0 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Host: www.sysinternals.com Accept: */* Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... HTTP/1.1 403 Access Forbidden Server: Microsoft-IIS/5.0 Date: Sat, 14 Aug 2004 15:58:58 GMT Content-Length: 4126 Content-Type: text/html Closing fd 1916 17:56:53 ERROR 403: Access Forbidden. FINISHED --17:56:53-- Downloaded: 0 bytes in 0 files Saving cookies to cookies.txt. Done saving cookies. --jeroen
