I hope that doesn't happen. While respecting robots.txt is not an absolute requirement, it is considered polite. I would not want the default behavior of wget to be considered impolite.
Mark Post -----Original Message----- From: Mauro Tortonesi [mailto:[EMAIL PROTECTED] Sent: Monday, August 08, 2005 7:43 PM To: Tony Lewis Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: robots.txt takes precedence over -p On Sunday 10 July 2005 09:52 am, Tony Lewis wrote: > Thomas Boerner wrote: > > Is this behaviour: "robots.txt takes precedence over -p" a bug or a > > feature? > > It is a feature. If you want to ignore robots.txt, use this command > line: > > wget -p -k www.heise.de/index.html -e robots=off hrvoje was thinking of changing the default behavior of wget to ignore the robots standard in the next releases. -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi http://www.tortonesi.com University of Ferrara - Dept. of Eng. http://www.ing.unife.it Institute for Human & Machine Cognition http://www.ihmc.us GNU Wget - HTTP/FTP file retrieval tool http://www.gnu.org/software/wget Deep Space 6 - IPv6 for Linux http://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
