Art Pollard wrote
This just came in from SearchEngineWatch.
I think it is pertinent to this discussion
It's pertinent AND impertinent! ;-)
LookSmart says that companies have been complaining that LookSmart's
tracking system will show a large number of clicks to their site which
Are you suggesting a robot is checking that string against
its UA??? I find
that hard to believe, but assuming that is the case such a
robot would be
allowed unrestricted access to looksmart.com, including all
their Pay Per
Click (PPC) URLs. I'm thinking that many robots are reading
On Wednesday, May 29, 2002 5:27 PM Walter Underwood wrote
As for your underlying question, I expect that you won't get a useful
answer. People at search engines really don't talk about how they
detect spammers and hostile bots. Once a technique is public, it is
dead.
I'm suggesting that
Rasmus Mohr writes:
Yes, that would be the case. For some unknown reason Looksmart allows
recognized robots/crawlers/spider and other non-standard user-agents
unlimited access according to the the robots.txt - all others are excluded.
I'd guess the weird looking java user-agent
PM
To: [EMAIL PROTECTED]
Subject: [Robots] Re: Looksmart's robots.txt file
Rasmus Mohr writes:
Yes, that would be the case. For some unknown reason
Looksmart allows
recognized robots/crawlers/spider and other non-standard
user-agents
unlimited access according
It seems to me that Looksmart is doing the right thing. Excluding
user-agents named Due to a deficiency in Java it's not currently possible
to set the User-Agent. will exclude all Java-based browsers unable to set
the user-agent property using the java.net.URLConnection.setRequestProperty
Rasmus wrote
It seems to me that Looksmart is doing the right thing. Excluding
user-agents named Due to a deficiency in Java it's not currently possible
to set the User-Agent. will exclude all Java-based browsers unable to
set
the user-agent property using the
--On Wednesday, May 29, 2002 04:49:22 PM +0100 Alan Perkins
[EMAIL PROTECTED] wrote:
Are you suggesting a robot is checking that string against its UA???
Yes. The user-agent line is probably an in-joke from a Perl programmer.
Also, I expect that it is very old. The specs seem to be in
