Thanks all! Especially for the tutorial link. I shouldn't see anymore 404's
for that file now.
Tom Fosson
At 10:24 PM 9/6/98 -0700, you wrote:
>> >> Everytime I submit my site to search engines, I start seeing a lot of
>> >> 404's in my logs for a file called 'robot.txt'. It's my
understanding that
>
>> > It tells robots where not to tread. Here is a version of one.
>>
>> It tells "GOOD" robots where not to tread. I had hotbot index 20
>> domains of mine that I didn't want it indexing.
>
> Yes, I gripe whenever a robot strays over the line. To determine
>that, I keep robot trap URL's in the various directories. These are URL's
>to non-existent files, (such as klank.html,) with no text between the A
>HREf and the /A end tag. If you see it in an access trace, you know that
>thing is a robot.
>
> My current problem is that there are many engines which, upon seeing
>a robots.txt file, just give up and don't come back.
>
>[EMAIL PROTECTED] ------------------ [EMAIL PROTECTED]
----------------------------------------
250 Megs of Web Space - Unlimited Autoresponders
Unlimited email aliases - Personal CGI-BIN Directory
MIVA programming scripting - over $650 Bonus pkg
T3 connection - Detailed tracking & referral reports
Free Domain Name hosting - NO SETUP FEE
Fantastic Business Opportunity - More - MUCH MORE!
Register to win a FREE color scanner while there!
http://www.moreprofits.com/cgi-bin/view.cgi/346/
----------------------------------------
Websites designed and published by Cyberspace Publishing
5515 N 7th Street, Ste 5-552, Phoenix, Arizona 85014
[EMAIL PROTECTED] * http://www.amug.org/~cyberpub/
____________________________________________________________________
--------------------------------------------------------------------
Join The Web Consultants Association : Register on our web site Now
Web Consultants Web Site : http://just4u.com/webconsultants
If you lose the instructions All subscription/unsubscribing can be done
directly from our website for all our lists.
---------------------------------------------------------------------
