> But making a search on the string "This page > was illegitimately indexed" reveals that most > legitimate robots have found it: Netsacpe, > Google, AOL, Compuserve,... you name it. > > So is it really working ? > IMO it is not safe to ban any robot on that only > basis.
I can tell you with absolute certainty that Google obeys robots.txt. So, I suspect that you've got a problem on your end. Did you add the changes to robots.txt after the file was crawled? Google requests robots.txt once prior to crawling your site, then once again for each recrawl. Does an HTTP request for robots.txt return a 200 status code (as opposed to a 301/302 redirect? As an alternative to robots.txt, you can include the appropriate META "robots" tag in the specific pages you want to exclude. Dave Watts, CTO, Fig Leaf Software ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Check out the new features and enhancements in the latest product release - download the "What's New PDF" now http://download.macromedia.com/pub/labs/coldfusion/cf8_beta_whatsnew_052907.pdf Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:292190 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
