> Art Pollard wrote > This just came in from SearchEngineWatch. > I think it is pertinent to this discussion ....
It's pertinent AND impertinent! ;-) > LookSmart says that companies have been complaining that LookSmart's > tracking system will show a large number of clicks to their site which > cannot be verified when the companies reconcile against their own traffic > logs. LookSmart says its "batch processing" of clicks before reporting is > to blame. Most people are measuring over a month, not three days. Some Webmasters are saying that Looksmart are reporting more clicks in a month than they are seeing in total - i.e. from all sources. Webmasters are quite capable of reading logs, they've been doing it for years. However, I'm not trying to bash Looksmart, I'm just trying to get to the bottom of the issue. I thought some wise list members may be able to help. IMO the way Looksmart listings are syndicated into their partners' search listings makes it near impossible for Looksmart to track random clicks from bots crawling their partners' search results when those bots are masquerading as browsers. They are very dependent on their partners' bot-detection capabilities. Maybe an example will help to illustrate the point. 1) Search on MSN for "search mechanics" (our product site) http://search.msn.com/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=search+mechanic s 2) the first page of results is headed up by a Looksmart listing, to http://www.looksmart.com/og/pr=cdd;cp=14;ci=15452;li=60749607;ed=20010618;ct =20020530|http://www.searchmechanics.com/ (Long URL - sorry!) The remainder of the page contains no URLs from Looksmart. 3) A bot masquerading as a browser and crawling links on the MSN results page would retrieve the above URL from Looksmart. 4) The HTTP response from the above Looksmart URL is a 302 to the advertiser's site (in this case www.searchmechanics.com). The bot may ignore the 302 and the advertiser will never see the visit. ***That's a reason why Looksmart would see more clicks than the advertisers.*** 5) OTOH the bot may follow the 302 but not supply the referrer in its HTTP request to the advertiser's site. *** That would be another reason why clicks at Looksmart could not be reconciled by advertisers. *** The situation is that some webmasters are seeing more hits reported by Looksmart than they are seeing in total, others are seeing more hits reported by Looksmart than they are seeing referrers from Looksmart. Real browsers do follow 302s and do report referrers. That's what is making me suspect bots as bot behaviour can explain both situations whereas browser behaviour cannot. I think putting it down to advertisers not being able to read their web logs correctly is a bit thin, but neither am I suggesting that Looksmart can't count! I'm suggesting an alternative explanation. There are bots that crawl search results pages, masquerade as browsers, and are run from many different IP addresses around the world. I'm still looking for someone to support or quash the theory. :-) Alan Perkins CTO, e-Brand Management Limited http://www.ebrandmanagement.com/
