yeosteve wrote: > If you go to google.com/robots.txt you can see all the folders they > don't want search engines to look at, so presumably if I was wearing > my black hat, I would start with this list of folders when looking for > private information. > http://www.robotstxt.org/faq/nosecurity.html
Q: Surely listing sensitive files is asking for trouble? A: The real answer is that /robots.txt is not intended for access control, so don't try to use it as such. *Think of it as a "No Entry" sign, not a locked door.* If you have files on your web site that you don't want unauthorized people to access, then configure your server to do authentication, and configure appropriate authorization. Basic Authentication has been around since the early days of the web (and in e.g. Apache on UNIX is trivial to configure). Modern content management systems support access controls on individual pages and collections of resources. > Does anyone know of a way to prevent this file being read by browsers > while still allowing search engines access to it? Even as I write > that, it seems like a Very Dumb question. > Google is a fantastic tool for black hats to find vulnerabilities, so making this information visible only to search engines isn't really helpful. Some quite innocuous Google Alerts I have set up, which were intended to keep an eye on my favourite tools in NZ, have reported vulnerable sites to me. And I don't even have a cool hat like Brenda. --~--~---------~--~----~------------~-------~--~----~ NZ PHP Users Group: http://groups.google.com/group/nzphpug To post, send email to [email protected] To unsubscribe, send email to [EMAIL PROTECTED] -~----------~----~----~----~------~----~------~--~---
