Thank you. I had been wondering; I unfortunately have to rebuild my web site tomorrow, but I think I'll make this new site use W3C instead of the IIS format.
From: [email protected] [mailto:[email protected]] On Behalf Of Ken Schaefer Sent: Friday, July 19, 2013 8:23 PM To: ozDotNet Subject: RE: [OT] Search engines (not a complaint) W3C is the standard. IIS defaults to using that. The IIS and NCSA format options are mainly there for backwards compatibility reasons. Cheers Ken From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Katherine Moss Sent: Saturday, 20 July 2013 12:02 AM To: ozDotNet Subject: RE: [OT] Search engines (not a complaint) Speaking of IIS logs, what is the most commonly used log format? I seem to be able to understand W3C more so than I can IIS; W3C seems to make it clearer what it going on. Anyone else agree with me? (though sooner rather than later, I won't have to worry about hand analysis because I'll hopefully install the free edition of SmarterStats on my server.) From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Greg Keogh Sent: Friday, July 19, 2013 3:46 AM To: ozDotNet Subject: [OT] Search engines (not a complaint) Just a bit of Friday technical trivia that might warn others ... I analysed all of the 2013 data so far from my IIS web logs and found the following interesting facts: robots.txt was read 11270 times My total sent bytes was 10.2GB of which 5.3GB was searching engines and bots 143374 requests came from bots (%46 of the total) So %52 of all traffic out of my web site was food for bots. I think this is an extraordinary volume of data and I have declared war on them. I found a sample robots.txt file on the web which looks quite comprehensive, then I added others ones I found in my logs. I installed the "IP Address and Domain Restrictions" Role to IIS 7.5 so I can totally block the worst offenders (I'm looking at you Ezooms and GoogleImages!). It will be interesting to see how my web request stats change over the coming months. I think this "search engine" zoo is now well out of control in the wild. So I've got charities and market researchers pestering me on the phone all day while my web server is pestered with a flood of bots. Greg
