--- Matthew Meadows <[EMAIL PROTECTED]> wrote: > I agree with Walter.
So do I, partially. :) > There's a lot of variables that should have > been > considered for this new value. If nothing else the specification > should > have called for the time in milliseconds, or otherwise allow for > fractional seconds. I disagree that level of granularity is needed. See my earlier email. > In addition, it seems a bit presumptuous for Yahoo > to think that they can force a de facto standard just by implementing > it first. That's how things work in real life. Think web browsers 10 years ago and various Netscape, then IE extensions. Now lots of them are considered standard. > With this line of thinking webmasters would eventually be > required to update their robots.txt file for dozens of individual > bots. In theory, yes. In reality, I agree with Walter, this extension will prove to be as useless as "<blink>", and will therefore not be supported by any big crawlers. > It's hard enough to get them to do it now for the general case, this > additional fragmentation is not going to make anybody's job easier. > Is > Google going to implement their own extensions, then MSN, AltaVista, > and AllTheWeb? Not likely. In order for them to remain competitive, they have to keep fetching web pages at high rates. robots.txt only limits them. I can't think of an extension to robots.txt that would let them do a better job. Actually, I can. :) > Finally, if we're going to start specifying the criteria for > scheduling, let's consider some other alternatives, like preferred > scanning windows. Same as crawl-delay - everyone would want crawlers to visit their sites at night, which would saturate crawlers' networks, so search engines won't push that extension. (actually, big crawlers run from multiple points around the planet, so maybe my statement is flawed) Otis > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] > On Behalf Of Walter Underwood > Sent: Friday, March 12, 2004 3:37 PM > To: Internet robots, spiders, web-walkers, etc. > Subject: Re: [Robots] Yahoo evolving robots.txt, finally > > > --On Friday, March 12, 2004 6:46 AM -0800 [EMAIL PROTECTED] > wrote: > > > > I am surprised that after all that talk about adding new semantic > > elements to robots.txt several years ago, nobody commented that the > > > new Yahoo crawler (former Inktomi crawler) took a brave step in > that > > direction by adding "Crawl-delay:" syntax. > > > > http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html > > > > Time to update your robots.txt parsers! > > No, time to tell Yahoo to go back and do a better job. > > Does crawl-delay allow decimals? Negative numbers? Could this spec be > a > bit better quality? The words "positive integer" would improve things > a > lot. > > Sigh. It would have been nice if they'd discussed this on the list > first. "crawl-delay" is a pretty dumb idea. Any value over one second > means it takes forever to index a site. Ultraseek > has had a "spider throttle" option to add this sort of delay, but it > is > almost never used, because Ultraseek reads 25 pages from one site, > then > moves to another. There are many kinds of rate control. > > wunder > -- > Walter Underwood > Principal Architect > Verity Ultraseek > > _______________________________________________ > Robots mailing list > [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots > _______________________________________________ > Robots mailing list > [EMAIL PROTECTED] > http://www.mccmedia.com/mailman/listinfo/robots _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
