On Tue, Jun 3, 2008 at 6:05 PM, Avi Rappoport <[EMAIL PROTECTED]> wrote:
> New REP is supported by Yahoo, Google and Microsoft, adds ALLOW > directive, wildcards, sitemaps, and more stuff to META tags. > > <http://www.ysearchblog.com/archives/000587.html> > < > http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html > > > > I'm wondering how many of the site/intranet search engines are nimble > enough to support these changes. > > Also, why <http://robotstxt.org> doesn't seem to be involved and > there's no neutral protocol web site... That site has described this mailing list as defunct for a long time, even though it never was... and my emails to them don't seem to produce any results. I'm not sure there's anyone home there. I have been very focused for the last few weeks on identifying spambots because we've seen a big increase in them lately... so I'm particularly interested in anything that helps us maintain white-lists of well-behaved bots. Nick -- Nick Arnett [EMAIL PROTECTED]
_______________________________________________ Robots mailing list Robots@mccmedia.com http://www.mccmedia.com/mailman/listinfo/robots
