But if you can't be found on a search engine, who's going to come to your site? Word of mouth isn't how you get most traffic. If you cut out the 52% of search engine bots, the other 48% won't know you exist.
On Fri, Jul 19, 2013 at 5:46 PM, Greg Keogh <[email protected]> wrote: > Just a bit of Friday technical trivia that might warn others ... > > I analysed all of the 2013 data so far from my IIS web logs and found the > following interesting facts: > > robots.txt was read 11270 times > My total sent bytes was 10.2GB of which 5.3GB was searching engines and > bots > 143374 requests came from bots (%46 of the total) > > So %52 of all traffic out of my web site was food for bots. I think this > is an extraordinary volume of data and I have declared war on them. I found > a sample robots.txt file on the web which looks quite comprehensive, then I > added others ones I found in my logs. I installed the "IP Address and > Domain Restrictions" Role to IIS 7.5 so I can totally block the worst > offenders (I'm looking at you Ezooms and GoogleImages!). > > It will be interesting to see how my web request stats change over the > coming months. I think this "search engine" zoo is now well out of control > in the wild. > > So I've got charities and market researchers pestering me on the phone all > day while my web server is pestered with a flood of bots. > > Greg > -- Meski http://courteous.ly/aAOZcv "Going to Starbucks for coffee is like going to prison for sex. Sure, you'll get it, but it's going to be rough" - Adam Hills
