I am surprised that after all that talk about adding new semantic
elements to robots.txt several years ago, nobody commented that the new
Yahoo crawler (former Inktomi crawler) took a brave step in that
direction by adding Crawl-delay: syntax.
--- Matthew Meadows [EMAIL PROTECTED] wrote:
I agree with Walter.
So do I, partially. :)
There's a lot of variables that should have
been
considered for this new value. If nothing else the specification
should
have called for the time in milliseconds, or otherwise allow for
fractional
