I just tried creating a web server locally.
|I tried creating a web server locally putting robots.txt in there and using
wget and it didn't work
http://pastebin.com/raw.php?i=kt1mV2af
C:\rwget 127.0.0.1:56
2012-03-16 19:45:32 (20.0 KB/s) - `index.html' saved [3/3] C:\rwget
I think you're misunderstanding what was supposed to happen.
The robots.txt file is only followed for links that wget is
automatically following. This means (a) wget has to be in
recursive-descent mode (-r or -m), and (b) it only applies to links that
weren't explicitly requested by the user. In
