--On Friday, March 12, 2004 6:46 AM -0800 [EMAIL PROTECTED] wrote: > > I am surprised that after all that talk about adding new semantic > elements to robots.txt several years ago, nobody commented that the new > Yahoo crawler (former Inktomi crawler) took a brave step in that > direction by adding "Crawl-delay:" syntax. > > http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html > > Time to update your robots.txt parsers!
No, time to tell Yahoo to go back and do a better job. Does crawl-delay allow decimals? Negative numbers? Could this spec be a bit better quality? The words "positive integer" would improve things a lot. Sigh. It would have been nice if they'd discussed this on the list first. "crawl-delay" is a pretty dumb idea. Any value over one second means it takes forever to index a site. Ultraseek has had a "spider throttle" option to add this sort of delay, but it is almost never used, because Ultraseek reads 25 pages from one site, then moves to another. There are many kinds of rate control. wunder -- Walter Underwood Principal Architect Verity Ultraseek _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
