What would be nice if the companies that create the likes of IIS and Apache and others, got together and defined a protocol at the server level, then they can request what they would like and if the server refuses them, then stiff shit jack.
I like rewrite rules for this purpose, and it maybe best at the firewall level, but any that disobey the robots.txt file is placed into the rewrite rules and I forward them to a sever 404 error. -- Regards, Andrew Scott WebSite: http://www.andyscott.id.au/ Google+: http://plus.google.com/108193156965451149543 On Wed, May 9, 2012 at 12:20 AM, <> wrote: > > >>this is the official way to do it. > http://www.robotstxt.org/ > > The problem with Robots.txt is that it is only obeyed by well behaved bots. > Well behaved bots won't cause traffic problem, even if they are a dozen > sucking your site in the same time. > I've seen bad bots reading robots.txt and immediately request all > exclusions. > Personaly, I use robots.txt to forbid a directory containing a trap for > bad bots and shut the door to them. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:351059 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
