Re: [Bug-wget] robots.txt seemingly ignored

2018-05-15 Thread Daniel Feenberg
Thank you. Updating to 1.19 fixed the problem. The version 1.12 came from the Scientific Linux v6 repository. I didn't realize it was so old. Installing 1.19 was easy - just configure;make;make install Thanks again. Daniel Feenberg On Tue, May 15, 2018 at 5:34 AM, Darshit Shah wrote: > Hi, > >

[Bug-wget] Bug-wget mailing list

2018-05-15 Thread Graeme
I opened the mailing list archives. http://lists.gnu.org/archive/html/bug-wget/ I searched for Windows, looking for information on the latest Wget for windows version. I get 1091 documents but they are in order of score.  I change the sort to by date and click search. In fact the heading of

[Bug-wget] Wget for WIndows download

2018-05-15 Thread Graeme
I am trying to find the up to date version of Wget for Windows. On https://www.gnu.org/software/wget/faq.html#download there are 4 links. Sourceforge and Bart Puype links seem to be dead links. Christopher Lewis' link times out. Jernej Simoncic's link works but the files don't seem to have ver

Re: [Bug-wget] robots.txt seemingly ignored

2018-05-15 Thread Darshit Shah
Hi, You are using a very old version of Wget. v1.12 was released in 2009 if I remember correctly. The current version of Wget doesn't seem to have any issues with the parsing of that robots.txt. I just tried it locally and it downloads no files at all. Please update your version of Wget. * Da