On Mon, 12 Apr 1999, Gilles Detillieux wrote:
>According to Chris Lott:
>> >ht://Dig follows the robots exclusion standard. Create a robots.txt
>> >file in your server root directory (it is also required by most www
>> >search engines, which won't index your site correctly if this file
>> >is not present).
>>
>> I thought this file was only necessary if one wanted to EXclude files, so
>> most of my sites don't have it. Where can I get more information on this
>> file and its format, etc?
>
>I believe you're right, Chris. If the robots.txt file is missing, robots
>should assume no restrictions or exclusions. See the Web Robots Pages:
>
> http://info.webcrawler.com/mak/projects/robots/robots.html
You're right Gilles.. they _should_. However, I experienced that /some/
robots only do the index in that case.. again /some/ look at the robots
tag there and go farther if it is present.. if not those critters simply
refuse to do their job.
Thus, having a robots.txt avoids these problems. Furthermore it helps
keeping down the number of nasty 404 entries in the server log files.
That's better than an Aspirin for most of the webmasters ,-)
ciao,
Torsten
--
InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstra�e 14 Tel: +49-4101-403605
D-25474 Ellerbek Fax: +49-4101-403606
E-Mail: [EMAIL PROTECTED] Internet: http://www.inwise.de
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the SUBJECT of the message.
