I just tried creating a web server locally.
|I tried creating a web server locally putting robots.txt in there and using
wget and it didn't work
http://pastebin.com/raw.php?i=kt1mV2af
C:\rwget 127.0.0.1:56
2012-03-16 19:45:32 (20.0 KB/s) - `index.html' saved [3/3] C:\rwget
http://stackprinter.appspot.com/export?question=282329format=HTMLservice=stackoverflowlinktohome=false
I've been looking at downloading a site that's on archive.org
I don't have the site in
front of me now but here are two example pages showing the kind of structure
i'm working with. Notice the website is spread in various directories by
archive.org
