Duke Hillard wrote:
worcester wrote:
I wonder if the issue is bigger than MSNBot. The list that you mentionedI was looking at the file here, http://wadsack-allen.com/products/robot-list.html
and noticed the msnbot wasn't listed. Am I missing something, or is the file?
(http://wadsack-allen.com/products/robot-list.html) contains 146 items, but
it also points to another list (http://www.robotstxt.org/wc/active/html/) that
contains 297 items (including MSNBot). Judging from the URL, I imagine
that Jeremy Wadsack maintains the list that you mentioned. If so, I am sure
that he will comment on the issue when he gets a spare moment or two.
Well, first of all the script does not bother to include the default robots listed in the analog.cfg that ships with Analog, namely:
REGEXPI:spider
REGEXPI:crawler
Googlebot*
Infoseek*
Scooter*
Slurp*
Ultraseek*
Which could make up for a few of those missing 151 items.
However, I also realized that it was only looking at robots with useragents of one word. I have fixed that and the list is now more inclusive. Oh, and it's sorted too.
Thanks,
-- Jeremy Wadsack Seven Simple Machines
+------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | Digest version: http://lists.isite.net/listgate/analog-help-digest/ | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
