I use the --spider option a lot and don't have trouble with most sites. When using the --spider option for the Mozilla website I get a 500 error response. Without the --spider option I don't receive the problem. Any guesses?
$ wget --debug --spider www.mozilla.org DEBUG output created by Wget 1.9-beta on aix4.3.3.0. --14:43:09-- http://www.mozilla.org/ => `index.html' Resolving www.mozilla.org... done. Caching www.mozilla.org => 207.200.81.215 Connecting to www.mozilla.org[207.200.81.215]:80... connected. Created socket 3. Releasing 20016c58 (new refcount 1). ---request begin--- HEAD / HTTP/1.0 User-Agent: Wget/1.9-beta Host: www.mozilla.org Accept: */* Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... HTTP/1.1 500 Server Error Server: Netscape-Enterprise/3.6 Date: Tue, 17 Jun 2003 18:43:09 GMT Content-length: 305 Content-type: text/html Connection: keep-alive Found www.mozilla.org in host_name_addresses_map (20016c58) Registered fd 3 for persistent reuse. Closing fd 3 Releasing 20016c58 (new refcount 1). Invalidating fd 3 from further reuse. 14:43:10 ERROR 500: Server Error. $ wget www.mozilla.org --14:43:42-- http://www.mozilla.org/ => `index.html' Resolving www.mozilla.org... done. Connecting to www.mozilla.org[207.200.81.215]:80... connected. HTTP request sent, awaiting response... 200 OK Length: 15,839 [text/html] 100%[====================================>] 15,839 22.65K/s 14:43:43 (22.65 KB/s) - `index.html' saved [15839/15839]
