Robots = off directive

2004-02-16 Thread chatiman
Hello, I'm trying to download a robots.txt protexted directory and I'm having the following problem: - wget downloads the files but delete them after they are downloaded with the following :message (translated from french): Destroyed file because it must be rejected How can I prevent this ?

Re: Robots = off directive

2004-02-16 Thread Hrvoje Niksic
chatiman [EMAIL PROTECTED] writes: I'm trying to download a robots.txt protexted directory and I'm having the following problem: - wget downloads the files but delete them after they are downloaded with the following :message (translated from french): Destroyed file because it must be

Re: Robots = off directive

2004-02-16 Thread patrick robinson
Hello Hrvoje, On 16-Feb-04, you wrote: chatiman [EMAIL PROTECTED] writes: I'm trying to download a robots.txt protexted directory and I'm having the following problem: - wget downloads the files but delete them after they are downloaded with the following :message (translated from

Re: Robots = off directive

2004-02-16 Thread Hrvoje Niksic
patrick robinson [EMAIL PROTECTED] writes: That message has nothing to do with robots.txt, it means that you have rejected the file using the `-R' or equivalent option. Here you go again with this IMHO stupid implemented option. Why thank you. I'm using it too but on some suffixes it acts