I have a robots.txt tutorials at perl2000.com/Tutorials/
Mainly it just tels the spider what it can and can not do. Most use it
there are some that ignore it still.
Chad
Cyberspace Publishing wrote:
>
> Everytime I submit my site to search engines, I start seeing a lot of
> 404's in my logs for a file called 'robot.txt'. It's my understanding that
> search engines look for that file for a list of pages to include or exclude
> in their indexing. Does anyone know where I can find information on
> the formatting, creation and locating of such a file? Thanks.
>
> Tom Fosson
>
> ----------------------------------------
> 250 Megs of Web Space - Unlimited Autoresponders
> Unlimited email aliases - Personal CGI-BIN Directory
> MIVA programming scripting - over $650 Bonus pkg
> T3 connection - Detailed tracking & referral reports
> Free Domain Name hosting - NO SETUP FEE
> Fantastic Business Opportunity - More - MUCH MORE!
> Register to win a FREE color scanner while there!
>
> http://www.moreprofits.com/cgi-bin/view.cgi/346/
> ----------------------------------------
> Websites designed and published by Cyberspace Publishing
> 5515 N 7th Street, Ste 5-552, Phoenix, Arizona 85014
> [EMAIL PROTECTED] * http://www.amug.org/~cyberpub/
>
> ____________________________________________________________________
> --------------------------------------------------------------------
> Join The Web Consultants Association : Register on our web site Now
> Web Consultants Web Site : http://just4u.com/webconsultants
> If you lose the instructions All subscription/unsubscribing can be done
> directly from our website for all our lists.
> ---------------------------------------------------------------------
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Chad D. Zimmerman
Email: [EMAIL PROTECTED] [EMAIL PROTECTED]
Webmaster: Perl Script Archive http://www.perl2000.com
Personal: http://www.brainfreeze.net/wildone/
ICQ #: 10875718
____________________________________________________________________
--------------------------------------------------------------------
Join The Web Consultants Association : Register on our web site Now
Web Consultants Web Site : http://just4u.com/webconsultants
If you lose the instructions All subscription/unsubscribing can be done
directly from our website for all our lists.
---------------------------------------------------------------------
