On 17/10/2009 09:08, Claude Schneegans wrote: > robots.txt file may be useful for well behaved robots like Google or > Yahoo, but bad bot disguise themselves > like "Explorer" or "Mozilla" and they do not comply to robots.txt > directives. >
that's exactly why i also said "of course, not all robots obey these rules - if this bot does not then you can consider banning it completely using the options you outlined." -- Azadi Saryev Sabai-dee.com http://www.sabai-dee.com/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:327292 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
