I recently downloaded and ran the security tool Nessus (www.nessus.com) Interestingly, Nessus reports the existance of robots.txt as a security "vulnerability" (one step worse than a "warning")
I wondered what the robot community might have to say on the topic. Quote: "Vulnerability found on port www (80/tcp) Some Web Servers use a file called /robot(s).txt to make search engines and any other indexing tools visit their WebPages more frequently and more efficiently. By connecting to the server and requesting the /robot(s).txt file, an attacker may gain additional information about the system they are attacking. Such information as, restricted directories, hidden directories, cgi script directories and etc. Take special care not to tell the robots not to index sensitive directories, since this tells attackers exactly which of your directories are sensitive. Risk factor : Medium" Andrew Daviel TRIUMF
