Walter Underwood wrote:
Nah, they would have e-mailed me directly by now. I used to work
with them at Inktomi.
How about dropping them an e-mail to invite them here?
Yahoo limits crawler access to its own site. I haven't tried in the
last 9 or 10 months, but the way it was back then, if you
points around the planet, so maybe my statement is flawed)
Otis
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
On Behalf Of Walter Underwood
Sent: Friday, March 12, 2004 3:37 PM
To: Internet robots, spiders, web-walkers, etc.
Subject: Re: [Robots] Yahoo evolving
--On Saturday, March 13, 2004 2:22 AM -0800 [EMAIL PROTECTED] wrote:
Does crawl-delay allow decimals?
You think people really want to be able to tell a crawler to fetch a
page at most every 5.6 seconds, and not 5?
0.5s would be useful. Ultraseek has used a float for the delay for
the past
robots, spiders, web-walkers, etc.
Subject: RE: [Robots] Yahoo evolving robots.txt, finally
--- Matthew Meadows [EMAIL PROTECTED] wrote:
I agree with Walter.
So do I, partially. :)
There's a lot of variables that should have
been
considered for this new value. If nothing else
--On Friday, March 12, 2004 6:46 AM -0800 [EMAIL PROTECTED] wrote:
I am surprised that after all that talk about adding new semantic
elements to robots.txt several years ago, nobody commented that the new
Yahoo crawler (former Inktomi crawler) took a brave step in that
direction by adding
: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Walter Underwood
Sent: Friday, March 12, 2004 3:37 PM
To: Internet robots, spiders, web-walkers, etc.
Subject: Re: [Robots] Yahoo evolving robots.txt, finally
--On Friday, March 12, 2004 6:46 AM -0800 [EMAIL PROTECTED] wrote:
I am
