Jeff, This will not prevent spiders from finding your site, there is some specific <head> tags that can do this fine without the robots.txt file. Also if you have back links to your site then search engines will find you this way as well. If you go through your logs on your web server you should find a large percentage of accesses attempting to find a robots.txt file, many engines look for it and some ignore it.
On a side note, if you are interested in spidering sites then you can make your own here: http://www.searchtools.com/robots/robot-code.html Peter Kaulback In the hour of 09:27 PM 7/25/2002 -0400, Jeff Dougherty spoke this: >Peter, >Won't this method also prevent search engines from spidering the site? >Jeff > >Intrepid Video & Electronics Be careful of your thoughts. >501 Luther Rd They may become your words >Harrisburg, PA 17111 any moment. > >Original Message ----- >From: "Peter Kaulback" <[EMAIL PROTECTED]> > >In the hour of 04:08 PM 7/24/2002 -0600, Roger Williams spoke this: > > >Hi workers, > >Do you know of any html tag or meta-tag which will stop robots from > >accessing a web page? I currently use the meta-tag "no index, no follow," > >but I know that few robots actually support that. The point is to deny > >harvesting or indexing of certain pages by robots on a web site. Anybody > >have any solutions? > > > >All the best, > >Roger > >Roger, you can add a robots text file to your site which can hamper robots >BUT if you are linked to any other sites the robots will find you. >More info can be found here: >http://www.robotstxt.org/wc/robots.html >http://www.searchengineworld.com/robots/robots_tutorial.htm > >And there is also the robots text file generator here: >http://www.webtoolcentral.com/webmaster/tools/robots_txt_file_generator/ > >Peter Kaulback ============= PCWorks Mailing List ================= Don't see your post? Check our posting guidelines & make sure you've followed proper posting procedures, http://pcworkers.com/rules.htm Contact list owner <[EMAIL PROTECTED]> Unsubscribing and other changes: http://pcworkers.com =====================================================
