--On Thursday, November 22, 2001 11:15 AM -0800 Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
>
> Yes, that's the problem. Robot writers don't care for robots.txt
> improvements (it would just slow their robots down), ..,
Not exactly.
1. robots.txt isn't used much. Around 5% of sites have one at all.
You can never double performance with a better robots.txt.
2. Fancy allow/disallow/rate specs are hard to specify, interpret,
update, and maintain. Site will get it wrong, robots will get it
wrong, etc. Note this, from the Robot.txt 2.0 proposal:
Unfortunately, there is no example code to parse Version 2.0
of the robot exclusion standard.
3. Rate control assumes robot behavior that may be very difficult
to control. Same for other semantic controls (time of day).
So yes, it would be slower, but this is much less important
than the first two reasons.
A fancy robots.txt is a bit like making a fancy turn signal
control for cars. If people won't signal in the first place,
it is the wrong thing to improve. And signals are advisory,
like robots.txt. It is the steering wheel that really does
the deed.
wunder
--
Walter Underwood
Senior Staff Engineer, Enterprise Search, Inktomi Corp.
http://search.inktomi.com/
--
This message was sent by the Internet robots and spiders discussion list
([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message
to "[EMAIL PROTECTED]".
