Message-
From: Mauro Tortonesi [mailto:[EMAIL PROTECTED]
Sent: Monday, August 08, 2005 8:35 PM
To: Post, Mark K
Cc: [EMAIL PROTECTED]
Subject: Re: robots.txt takes precedence over -p
On Monday 08 August 2005 07:30 pm, Post, Mark K wrote:
I hope that doesn't happen. While respecting robots.txt
On Sunday 10 July 2005 09:52 am, Tony Lewis wrote:
Thomas Boerner wrote:
Is this behaviour: robots.txt takes precedence over -p a bug or
a feature?
It is a feature. If you want to ignore robots.txt, use this command line:
wget -p -k www.heise.de/index.html -e robots=off
hrvoje was
08, 2005 7:43 PM
To: Tony Lewis
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: robots.txt takes precedence over -p
On Sunday 10 July 2005 09:52 am, Tony Lewis wrote:
Thomas Boerner wrote:
Is this behaviour: robots.txt takes precedence over -p a bug or a
feature?
It is a feature
unless I tell them otherwise.
Mark Post
-Original Message-
From: Mauro Tortonesi [mailto:[EMAIL PROTECTED]
Sent: Monday, August 08, 2005 8:35 PM
To: Post, Mark K
Cc: [EMAIL PROTECTED]
Subject: Re: robots.txt takes precedence over -p
On Monday 08 August 2005 07:30 pm, Post, Mark K wrote:
I
