Re: Robots = off directive
patrick robinson <[EMAIL PROTECTED]> writes: >> That message has nothing to do with robots.txt, it means that you >> have rejected the file using the `-R' or equivalent option. > > Here you go again with this IMHO stupid implemented option. Why thank you. > I'm using it too but on some suffixes it acts after downloading by > deleting the already downloaded file and on other suffixes it works > in advance. It works /a posteriori/ on HTML documents because they need to be downloaded to be examined for links. Otherwise something like `wget -r -A jpg URL' would not download anything because the index item is not an image. > But I'm only using version 1.8.2 maybe it has been changed in more > recent versions by now. Many things have improved since 1.8.2. I recommend upgrade.
Re: Robots = off directive
Hello Hrvoje, On 16-Feb-04, you wrote: > "chatiman" <[EMAIL PROTECTED]> writes: >> I'm trying to download a robots.txt protexted directory and I'm having the >> following problem: >> >> - wget downloads the files but delete them after they are downloaded with >> the following :message (translated from french): >> Destroyed because it must be rejected > That message has nothing to do with robots.txt, it means that you have > rejected the file using the `-R' or equivalent option. Here you go again with this IMHO stupid implemented option. I'm using it too but on some suffixes it acts after downloading by deleting the already downloaded file and on other suffixes it works in advance. I wonder why it not always rejects predefined files/suffixes in advance. I doesn't make much sense to download them and then delete them. But I'm only using version 1.8.2 maybe it has been changed in more recent versions by now. Regards Patrick Robinson
Re: Robots = off directive
"chatiman" <[EMAIL PROTECTED]> writes: > I'm trying to download a robots.txt protexted directory and I'm having the > following problem: > > - wget downloads the files but delete them after they are downloaded with > the following :message (translated from french): > Destroyed because it must be rejected That message has nothing to do with robots.txt, it means that you have rejected the file using the `-R' or equivalent option.
Robots = off directive
Hello, I'm trying to download a robots.txt protexted directory and I'm having the following problem: - wget downloads the files but delete them after they are downloaded with the following :message (translated from french): Destroyed because it must be rejected How can I prevent this ? Thanks PS: I'm using wget 1.8.1-6
