>>What would be nice if the companies that create the likes of IIS and Apache and others, got together and defined a protocol at the server level.
IMHO, the only rule that would be really useful would be to force Robots to 1. identify the web page were they discribe their purpose and what they are lookin for, 2. give a list of IP addresses or range they are crawling from so we can white or black list them. But asking them to obey robots.txt file is like relying on laws to keep burglars away from your house. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:351060 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
