I am surprised that after all that talk about adding new semantic elements to robots.txt several years ago, nobody commented that the new Yahoo crawler (former Inktomi crawler) took a brave step in that direction by adding "Crawl-delay:" syntax.
http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html Time to update your robots.txt parsers! Otis Gospodnetic _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
