Found a bug (sort of). When trying to get all the images in the directory below: http://www.netstate.com/states/maps/images/
It gives 403 Forbidden errors for most of the images even after setting the agent string to firefox's, and setting -e robots=off After a packet capture, it appears that the site will give the forbidden error if the Refferer is not exaclty correct. However, since wget actually uses the domain www.netstate.com:80 instead of without the port, it screws it all up. I've been unable to find any way to tell wget not to insert the port in the requesting url and referrer url. Here is the full command I was using: wget -r -l 1 -H -U "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" -e robots=off -d -nh http://www.netstate.com/states/maps/images/ -Reece
