>This is wrong.
>
>The spec [1] doesn't say anything about order being significant. One of the
>examples in the spec shows the robot "cybermapper" matching a user-agent
>line which is after a "*" entry.
>
>A robot which implements "first match" is not following the spec.
>
>The spec text in the proposed RFC [2] (never adopted) is more specific
>about this:
>
> These name tokens are used in User-agent lines in /robots.txt to
> identify to which specific robots the record applies. The robot
> must obey the first record in /robots.txt that contains a User-
> Agent line whose value contains the name token of the robot as a
> substring. The name comparisons are case-insensitive. If no such
> record exists, it should obey the first record with a User-agent
> line with a "*" value, if present.
>
>If you want to protect against this particular variety of robot
>mis-coding, go ahead. I don't know if there is any evidence of this
>behavior in the wild, so it might be a waste of time.
It may be wrong according to the specifications, but that's
what seemed to work when I was trying to get rid of a couple of
pesky spiders several years ago. With 'cut and paste', it isn't
going to take a second to move it to the end of the file. Try it.
It just might work for you.
Fred Atkinson
_______________________________________________
Robots mailing list
Robots@mccmedia.com
http://www.mccmedia.com/mailman/listinfo/robots
