At 09:10 PM 9/6/98 -0700, [EMAIL PROTECTED] wrote:
>>
>> Everytime I submit my site to search engines, I start seeing a lot of
>> 404's in my logs for a file called 'robot.txt'. It's my understanding that
>> search engines look for that file for a list of pages to include or exclude
>> in their indexing. Does anyone know where I can find information on
>> the formatting, creation and locating of such a file? Thanks.
>
> It tells robots where not to tread. Here is a version of one.
It tells "GOOD" robots where not to tread. I had hotbot index 20
domains of mine that I didn't want it indexing.
Matt Soffen
==============================================
Boss - "My boss says we need some eunuch programmers."
Dilbert - "I think he means UNIX and I already know UNIX."
Boss - "Well, if the company nurse comes by, tell her I said
never mind."
- Dilbert -
==============================================
____________________________________________________________________
--------------------------------------------------------------------
Join The Web Consultants Association : Register on our web site Now
Web Consultants Web Site : http://just4u.com/webconsultants
If you lose the instructions All subscription/unsubscribing can be done
directly from our website for all our lists.
---------------------------------------------------------------------
