> >> On some sites I cannot download files. > >> On one such site I found this file "robots.txt". > >> Is this file the cause for wget not downloading the files. > > Why not just put "robots=off" in your .wgetrc? > > Read the docs, people! That's why many hours were spent by persons > preparing, correcting, and keeping them up to date as best they can.
With all due respect, I *DID* read the documentation, which I quoted, and I attempted to find the latest version available. I didn't post the original question, and *MY* question still stands: if you must change the User-Agent value, what are some "legal" alternatives? If the User-Agent value change is deprecated in favour of robots=off, that's fine -- does the use of robots=off translate in source into a change in the value for User-Agent that is sent to the server? --Amy
