Here's a test case for the --spider option. perhaps helpful for documentation?
using wget on about 17,000 URLs (these are in the FSF/UNESCO Free Software Directory and are not by any means unique). out of these about 395 generate errors when run with the spider option (--spider) of the wget command. --spider uses an HTTP HEAD request rather than a GET interestingly when running again without the spider command (instead just using --delete-after) only 334 URLs generate errors. 61 of these URLs actually worked. most likely because the server only supported the HTTP GET request and choked on the HTTP HEAD request. not necessarily a worthy sample, but 61 out of 17000 URLs don't support wget's spidering technique. /a
