I just tried creating a web server locally. |I tried creating a web server locally putting robots.txt in there and using wget and it didn't work
http://pastebin.com/raw.php?i=kt1mV2af C:\r>wget 127.0.0.1:56 .... 2012-03-16 19:45:32 (20.0 KB/s) - `index.html' saved [3/3] C:\r>wget 127.0.0.1:56/robots.txt .... 2012-03-16 19:45:43 (175 KB/s) - `robots.txt' saved [26/26] C:\r>type robots.txt User-agent: * Disallow: / C:\r>
