> A client of ours recently had a security audit on their web > site. The audit recommended that we remove all 'disallow: /xyz/' > entries since a potential hacker could read the robots.txt file > and surmise which folders may be sensitive. > > Here's my question, if I remove all of the [disallow: /xyz/] > lines from the robots.txt file, how do I stop the search engines > from indexing those directories?
Unfortunately, this kind of information leakage is largely unavoidable. For most security audits, you may have to provide a case-by-case justification for each line within robots.txt explaining why the information leakage isn't important - for example, if the URL patterns in question don't actually lead to sensitive information. Very often, they don't. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ Fig Leaf Software provides the highest caliber vendor-authorized instruction at our training centers in Washington DC, Atlanta, Chicago, Baltimore, Northern Virginia, or on-site at your location. Visit http://training.figleaf.com/ for more information! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Discover CFTicket - The leading ColdFusion Help Desk and Trouble Ticket application http://www.houseoffusion.com/banners/view.cfm?bannerid=48 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:207883 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
