At 1:01 PM +0200 8/14/02, pp wrote: >I got robots.txt like this: > >User-agent: * >Disallow: /page > >This should disallow all robots index all pages within /page. >Right?
Nope, you should disallow an entire directory with a slash at the end, like this: /page/ Check your robots formatting using a checker: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi http://www.tardis.ed.ac.uk/%7Esxw/robots/check/ http://www.ukoln.ac.uk/web-focus/webwatch/services/robots-txt/ More information at <http://www.robotstxt.org/wc/exclusion-admin.html> <http://www.searchtools.com/robots/robots-txt.html> Avi -- Complete Guide to Search Engines for Web Sites and Intranets <http://www.searchtools.com> ------------------------------------------------------- This sf.net email is sponsored by: OSDN - Tired of that same old cell phone? Get a new here for FREE! https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390 _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
