At 8:54 PM -0700 6/3/08, Nick Arnett wrote:
>On Tue, Jun 3, 2008 at 6:05 PM, Avi Rappoport
><<mailto:[EMAIL PROTECTED]>[EMAIL PROTECTED]> wrote:
>
>New REP is supported by Yahoo, Google and Microsoft, adds ALLOW
>directive, wildcards, sitemaps, and more stuff to META tags.
>
><<http://www.ysearchblog.com/archives/000587.html>http://www.ysearchblog.com/archives/000587.html>
><<http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html>http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html>
>
>I'm wondering how many of the site/intranet search engines are nimble
>enough to support these changes.
>
>Also, why <<http://robotstxt.org>http://robotstxt.org> doesn't seem
>to be involved and
>there's no neutral protocol web site...
>
>
>That site has described this mailing list as defunct for a long
>time, even though it never was... and my emails to them don't seem
>to produce any results. I'm not sure there's anyone home there.
>
>I have been very focused for the last few weeks on identifying
>spambots because we've seen a big increase in them lately... so I'm
>particularly interested in anything that helps us maintain
>white-lists of well-behaved bots.
How about this, from the Big Three: "Each of us also supports Reverse
DNS based authentication of our crawler, and you can use this
validate the identity of any crawlers claiming a particular
user-agent."
Avi
--
Complete Guide to Search Engines for Web Sites and Intranets
<http://www.searchtools.com>
_______________________________________________
Robots mailing list
[email protected]
http://www.mccmedia.com/mailman/listinfo/robots
