I think the distinction needs to be make between what's sensitive and what you don't want to be indexed by a spider.
If the information is sensitive, it shouldn't be where a spider can get to it at all - and robots.txt is no protection. If the information just shouldn't be indexed, then robots.txt will work ... at least for well-behaved spiders. Chris Norloff ---------- Original Message ---------------------------------- From: "Che Vilnonis" <[EMAIL PROTECTED]> Reply-To: [email protected] Date: Fri, 27 May 2005 10:30:08 -0400 >A client of ours recently had a security audit on their web site. The audit >recommended that we remove all 'disallow: /xyz/' entries since a potential >hacker could read the robots.txt file and surmise which folders may be >sensitive. > >Here's my question, if I remove all of the [disallow: /xyz/] lines from the >robots.txt file, how do I stop the search engines from indexing those >directories? > >Thanks, Che > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Find out how CFTicket can increase your company's customer support efficiency by 100% http://www.houseoffusion.com/banners/view.cfm?bannerid=49 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:207863 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
