>>Have you confirmed that the robots have requested and successfully retrieved robots.txt (perhaps search the logs for the webserver)?
No, I do not trace reading of robots.txt. In principle good robot should read and honor it. Obviously, there is no absolutely good robot. I use Copernic for searching, and it will interrogate at least 15 search engines with the same string, so I find all occurreces in all af them. Some other bot trap use different warning strings, like "If you are a human: Uh-Oh. You just accessed the bad-robot trap." or "There is nothing here to see. So what are you doing here ?" It's amazing the number of pages in directories like /bot-trap that all engines report. This does not mean that all robots do not honor robots.txt, for instance, if only once, the HTTP server fails or takes too long to return the file, the the robot may be banned for life from your site. Too dangerous. So I'm presently thinking about some othe way to detect and ban bad robots. I'm especially looking for robots reading images, so it should not be too complicated using CF. Good robots like Google should not read images. -- _______________________________________ REUSE CODE! Use custom tags; See http://www.contentbox.com/claude/customtags/tagstore.cfm (Please send any spam to this address: [EMAIL PROTECTED]) Thanks. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| ColdFusion 8 - Build next generation apps today, with easy PDF and Ajax features - download now http://download.macromedia.com/pub/labs/coldfusion/cf8_beta_whatsnew_052907.pdf Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:292187 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
