Rasmus wrote
> It seems to me that Looksmart is doing the right thing. Excluding > user-agents named "Due to a deficiency in Java it's not currently possible > to set the User-Agent." will exclude all Java-based "browsers" unable to set > the user-agent property using the java.net.URLConnection.setRequestProperty > method. Are you suggesting a robot is checking that string against its UA??? I find that hard to believe, but assuming that is the case such a robot would be allowed unrestricted access to looksmart.com, including all their Pay Per Click (PPC) URLs. I'm thinking that many robots are reading looksmart.com, some with permission from robots.txt, and some without. I'm looking for reasons why advertisers can't reconcile clickthroughs with the figures provided by Looksmart. One suggestion is that if Looksmart aren't checking the User Agents of clients accessing the PPC URLs, that would be a reason why many more clicks were being seen by Looksmart than by advertisers. The robot either may not follow the redirect, or may follow without providing a referrer, or may be silently filtered by an advertiser's stats package because it is a robot not a human visitor. Another suggestion is that some robots will be masquerading as browsers, but still may not follow redirects or send a referrer allowing the clicks to be reconciled. I'm trying to gather likelihood on the possibility of each scenario, so I'm looking for a) how many robots, given www.looksmart.com/robots.txt, would read those looksmart.com PPC URLs? b) how many of those robots would be recognisable as robots, i.e. use a unique User Agent? Do we all agree that if a robot masquerades as a browser, ignores robots.txt and incurs clickthrough fees for advertisers, then the advertisers' beef (if any) should be with the robot owner rather than the PPC provider? But if a robot sends a recognisable UA, complies with robots.txt and advertisers still incur clickthrough fees, the advertisers' beef (if any) should be with the PPC provider? Alan Perkins CTO, e-Brand Management Limited http://www.ebrandmanagement.com/
