--- Matthew Meadows [EMAIL PROTECTED] wrote:
I agree with Walter.
So do I, partially. :)
There's a lot of variables that should have
been
considered for this new value. If nothing else the specification
should
have called for the time in milliseconds, or otherwise allow for
fractional
--On Saturday, March 13, 2004 2:22 AM -0800 [EMAIL PROTECTED] wrote:
Does crawl-delay allow decimals?
You think people really want to be able to tell a crawler to fetch a
page at most every 5.6 seconds, and not 5?
0.5s would be useful. Ultraseek has used a float for the delay for
the past
I'm standing firm on my suggestions. Adding a delay for crawlers is a
good idea in concept,
and allowing fractional seconds is a way for webmasters to request
reasonable constraints. Is
it such a stretch to allow a robot that you use to promote your business
unmitigated access to
your site, but
