> On Wednesday, May 29, 2002 5:27 PM Walter Underwood wrote
> As for your underlying question, I expect that you won't get a useful > answer. People at search engines really don't talk about how they > detect spammers and hostile bots. Once a technique is public, it is > dead. I'm suggesting that (IMO) there is a mis-match between the backroom skills that would be required to detect a hostile bot, and the one place where those backroom skills are displayed in public - the robots.txt file. I am looking for evidence to support or quash the theory that robots are one cause of the fact that advertisers cannot reconcile clickthroughs with those reported by Looksmart. > robots.txt is for cooperating robots. If a robot is not going to > cooperate, then robots.txt is just a file. If a bot lies about its > identity or intentions, that is clearly not "cooperating". Agreed. Looksmart's robots.txt is saying "if you're a cooperating bot, you can have unrestricted access, but if you're not a cooperating bot, you can't have access". In other words, the whole thing is redundant! A lot of "noise" could be removed by carefully restricting cooperating bots using robots.txt. If you are a search engine or directory that spends a great deal of time, effort and money building and maintaining an index, why allow bots to crawl that index? In particular, why allow cooperating bots to crawl your PPC listings, when you will then have to spend time and effort filtering out those hits? The theory I am trying to prove/disprove is that not all such hits are filtered. > Note that it is possible to have benign non-cooperating bots, like > a checker for handicapped accessbility that sends a particular browser > user-agent line (see http://www.cast.org/bobby/). Agreed again. Although IMO to be classified as benign a bot should at least obey instructions to User-Agent: *, otherwise it can cause some of the "side-effects" referred to by the original robots.txt spec. The side-effect quoted in the spec was voting. Causing a PPC payment is another side-effect. > On Thursday, May 30, 2002 8:20 AM Rasmus Mohr wrote > eh...beef? Apologies! In this context it means complaint or grievance. Alan Perkins CTO, e-Brand Management Limited http://www.ebrandmanagement.com/
