Richard,
You're absolutely right! But why does the Acme.Spider use that user-agent id? I can't find any reference to it in the source, i wonder.... > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On > Behalf Of [EMAIL PROTECTED] > Sent: Thursday, May 30, 2002 3:58 PM > To: [EMAIL PROTECTED] > Subject: [Robots] Re: Looksmart's robots.txt file > > > > Rasmus Mohr writes: > > Yes, that would be the case. For some unknown reason > Looksmart allows > > recognized robots/crawlers/spider and other non-standard > user-agents > > unlimited access according to the the robots.txt - all > others are excluded. > > I'd guess the weird looking "java" user-agent originates > from an Java > > application running on a platform/JVM unable to set the > user-agent property. > > The guys at Looksmart probably detected it in their logfiles... > > I don't think so. I think they just processed the web robots list > autocatically. In fact, that's what it says at the top of the > robots.txt file. If you look at > http://www.robotstxt.org/wc/active/html/contact.html you'll see where it comes from. > eh...beef? gripes, wrath, criticism, complaints, etc. General feeling of displeasure directed to some person or thing, Richard -------------------------------------------------------------- Rasmus T. Mohr Direct : +45 36 910 122 Application Developer Mobile : +45 28 731 827 Netpointers Intl. ApS Phone : +45 70 117 117 Vestergade 18 B Fax : +45 70 115 115 1456 Copenhagen K Email : mailto:[EMAIL PROTECTED] Denmark Website : http://www.netpointers.com "Remember that there are no bugs, only undocumented features." --------------------------------------------------------------
