> On some sites I cannot download files. > On one such site I found this file "robots.txt". > Is this file the cause for wget not downloading the files. > > Any way I could circumvent this and download this website.
Yep, you have to set the User-Agent value (which is an argument to wget) to "pretend" to be something the site will accept: typically Netscape or Internet Explorer. The ability to set this value was not present in some earlier versions of wget; someone more knowledgeable than I will know. However, the documentation for wget says that Netscape considers "faking" wget's identity to be that of Netscape/Mozilla "fraud." The documentation states "DO NOT" do this. So, then, what *are* some acceptable values for this field? Thanks, Amy
