Sorry for not getting back earlier I totally forgot about this. But I have never actually used the robots.txt file but I looked at their spec and you seem to have it correct. I also saw that html elements are also used in some cases.
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> I was looking at: http://www.robotstxt.org/wc/exclusion.html Hope that helped you some. ------------------------------------------ Ali Mesdaq Security Researcher II Websense Security Labs http://www.WebsenseSecurityLabs.com ------------------------------------------ -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Winterlight Sent: Wednesday, May 09, 2007 12:09 PM To: The Hardware List Subject: RE: [H] take off search list At 11:57 AM 5/9/2007, you wrote: >Robots.txt file. Google it it's a spec most search engines follow but >they don't have to. There are also other techniques like writing the >page with javascript instead of having it written when served. I can go >into details depending on if that robots.txt is sufficient. > >Thanks, >------------------------------------------ >Ali Mesdaq >Security Researcher II >Websense Security Labs >http://www.WebsenseSecurityLabs.com >------------------------------------------ Yes this is all I need, thanks. In fact, it is a personal site and it would be fine just to do the whole site. So if I just create a robot.txt file with User-agent: * Disallow: / in it and put it in the root directory.... will that be enough to prevent Google and most other sites from indexing me?
