>>When do you use a robots.txt file? >>What directories do you disallow?
In my case mainly two things: 1. normally, ie. to tell robots pages not worth indexing, mainly because the require some sort of login that robots won't be able to pass anyway; 2. I've declared a trap template in robots.txt file, if a robot still opens it, it means it does not comply, then it can be suspected to be just a mail tracker or so. I also have the possibility to CFABORT any robot I don't like, right in the application.cfm, although it is not done yet. -- _______________________________________ REUSE CODE! Use custom tags; See http://www.contentbox.com/claude/customtags/tagstore.cfm (Please send any spam to this address: [EMAIL PROTECTED]) Thanks. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:228987 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
