-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Micah Cowan wrote: > Nico Golde wrote: >> sometimes it would be nice to ignore robots.txt. What about adding an option >> to ignore this file? > > It has one. :) Check the info manual (the manpage is not the complete > documntation). You want -e robots=off
But, people often put Wget in the robots file (especially when they opt to do that rather than block by User-Agent) because they get hammered by it. Please be gracious, and use --wait or --limit-rate as appropriate to avoid overloading the server. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIsayW7M8hyUobTrERAow0AJ9GSt0ynfJTItmNt35gzzSaOn9V7wCeJwfK Kv5eFA1olqf8BMIp1qIjRzo= =4I8Z -----END PGP SIGNATURE----- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
