> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On
> Behalf Of Sean M. Burke
...
> E.g., http://www.robotstxt.org/wc/norobots.html says:
> <<User-agent [...] The robot should be liberal in interpreting
> this field.
> A case insensitive substring match of the name without version
> information
> is recommended.>>
>
> ...note the "without version information". Ditto the spec you
> cited, which
> says <<That is, the User-Agent (HTTP) header consists of one or
> more words,
> and the very first word is taken to be the "name", which is
> referred to in
> the robot exclusion files.>>
Ah, now I see your point. That does seem to be a problem, since apparently
version numbers were contemplated in User-Agent headers... Sounds like
something for the LWP author(s).
Or, a convenient excuse for a badly behaved robot... !
Nick
--
This message was sent by the Internet robots and spiders discussion list
([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message
to "[EMAIL PROTECTED]".
