> >> Everytime I submit my site to search engines, I start seeing a lot of
> >> 404's in my logs for a file called 'robot.txt'. It's my understanding that
> > It tells robots where not to tread. Here is a version of one.
>
> It tells "GOOD" robots where not to tread. I had hotbot index 20
> domains of mine that I didn't want it indexing.
Yes, I gripe whenever a robot strays over the line. To determine
that, I keep robot trap URL's in the various directories. These are URL's
to non-existent files, (such as klank.html,) with no text between the A
HREf and the /A end tag. If you see it in an access trace, you know that
thing is a robot.
My current problem is that there are many engines which, upon seeing
a robots.txt file, just give up and don't come back.
[EMAIL PROTECTED] ------------------ [EMAIL PROTECTED]
----------------------- IMAGINEERING --------------------------
----------------- Every mouse click, a Vote -------------------
---------- Do they vote For, or Against your pages? -----------
----- What people want: http://www.mall-net.com/se_report/ ----
---------------------------------------------------------------
--- Have you analyzed your viewer's footprints in the logs? ---
--- Webmaster's Resources: http://www.mall-net.com/webcons/ ---
--- Web Imagineering -- Architecture to Programming CGI-BIN ---
---------------------------------------------------------------
____________________________________________________________________
--------------------------------------------------------------------
Join The Web Consultants Association : Register on our web site Now
Web Consultants Web Site : http://just4u.com/webconsultants
If you lose the instructions All subscription/unsubscribing can be done
directly from our website for all our lists.
---------------------------------------------------------------------
