>>this is the official way to do it. http://www.robotstxt.org/
The problem with Robots.txt is that it is only obeyed by well behaved bots. Well behaved bots won't cause traffic problem, even if they are a dozen sucking your site in the same time. I've seen bad bots reading robots.txt and immediately request all exclusions. Personaly, I use robots.txt to forbid a directory containing a trap for bad bots and shut the door to them. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:351058 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
