This one time, at band camp, Peter Chubb wrote: > I'm a little cheesed off. In the last three months, people have > downloaded 9G per month from our website; search engines have > downloaded 21G per month. Only Google generated significant traffic > through search engine hits (and it downloaded less than the others, > too --- around 2G per month, as opposed to 10G for Yahoo, and 4G for > MSNbot). In other words, search engine indexing traffic was double > the actual traffic from www.gelato.unsw.edu,au.
Ever since Google started being _really_ fast with index updates, all the others have been trying to catch up. By "really fast" I mean my work launched a new campaign on Sunday morning and it was in Google's results by Monday afternoon. > Is there any good reason why I shouldn't block (or at least > significantly slow down) MSNbot, MJ12BOT, and Yahoo > Slurp! ??? Yahoo is particularly bad, crawling and downloading about > twice what the others do, and yet generating 1% of the hits that > Google generated for us. Depends how much you care about Yahoo's or MSN's referrals. In the consumer space I work in, Google still accounts for the vast majority of hits (>85%) and a similar proportion of sales. MSN + Live are around 10%, essentially through Microsoft's domination of people who don't know how to change their home page let alone install an alternative browser. Yahoo gets around 4%. For us, these are the people we target so it's very important. For you, it might be less important. You might want to look into the Crawl-delay extension to the robots.txt standard, which can limit by robot: http://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive -- Rev Simon Rumble <[EMAIL PROTECTED]> www.rumble.net The Tourist Engineer Geeks need vacations too. http://engineer.openguides.org/ "History teaches us that men and nations behave wisely once they have exhausted all other alternatives." - Abba Eban -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
