According to Chris Lott:
> >ht://Dig follows the robots exclusion standard. Create a robots.txt
> >file in your server root directory (it is also required by most www
> >search engines, which won't index your site correctly if this file
> >is not present).
>
> I thought this file was only necessary if one wanted to EXclude files, so
> most of my sites don't have it. Where can I get more information on this
> file and its format, etc?
I believe you're right, Chris. If the robots.txt file is missing, robots
should assume no restrictions or exclusions. See the Web Robots Pages:
http://info.webcrawler.com/mak/projects/robots/robots.html
--
Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]>
Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba Phone: (204)789-3766
Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the SUBJECT of the message.
