New REP is supported by Yahoo, Google and Microsoft, adds ALLOW directive, wildcards, sitemaps, and more stuff to META tags.
<http://www.ysearchblog.com/archives/000587.html> <http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html> I'm wondering how many of the site/intranet search engines are nimble enough to support these changes. Also, why <http://robotstxt.org> doesn't seem to be involved and there's no neutral protocol web site... Avi -- Complete Guide to Search Engines for Web Sites and Intranets <http://www.searchtools.com> _______________________________________________ Robots mailing list Robots@mccmedia.com http://www.mccmedia.com/mailman/listinfo/robots
