Rasmus Mohr writes:
> Yes, that would be the case. For some unknown reason Looksmart allows
> recognized robots/crawlers/spider and other non-standard user-agents
> unlimited access according to the the robots.txt - all others are excluded.
> I'd guess the weird looking "java" user-agent originates from an Java
> application running on a platform/JVM unable to set the user-agent property.
> The guys at Looksmart probably detected it in their logfiles...
I don't think so. I think they just processed the web robots list
autocatically. In fact, that's what it says at the top of the
robots.txt file. If you look at
http://www.robotstxt.org/wc/active/html/contact.html
you'll see where it comes from.
> eh...beef?
gripes, wrath, criticism, complaints, etc. General feeling of displeasure
directed to some person or thing,
Richard
