On Friday, February 11, 2005 7:14 PM [GMT], Ken Schweigert <[EMAIL PROTECTED]> wrote:
> Or ... to regenerate it at your convenience: > > > [EMAIL PROTECTED] tmp]$ wget http://www.robotstxt.org/wc/active/all.txt > [EMAIL PROTECTED] tmp]$ grep "robot-name:" all.txt | awk -F: '{print $2}' | > sed 's/^ *//g' | sort | awk '{print "ROBOTINCLUDE \"" $1 "*\""}' grep "robot-name:" or grep "robot-useragent:"? Aengus +------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.meer.net/mailman/listinfo/analog-help | | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
