Search engine spiders read that file to see which directories we don't want
them to index. I heard rumours that a certain search engine was ignoring the
file, and I wanted to verify that.
> On Wed, 10 Nov 1999, Michael Packer wrote:
>
> MP>>since your web site seems to be on a Unix box (at least netcom.ca is
> MP>>Unix) why
> MP>>not just logon to the server and do something like:
> MP>>
> MP>>grep robots.txt access_log
>
> What are they looking for when they request 'robots.txt'?
>
> -= Jim =-
>
> ----------------------------------------------------------------
> Jim's Linux-Operated Underground Bomb Shelter
> (Specializing in Hot Peppers and Mushrooms, and wayward kids)
> ----------------------------------------------------------------
>
> Tagline for Wednesday, November 10, 1999 at 16:50 PM:
>
> We secretly replaced the dilithium with Folger's Crystals...
>
> ----------------------------------------------------------------
> This Linux System has been up 973 hours
>
> My web page: http://www.idkent.com
> ----------------------------------------------------------------
>
> ------------------------------------------------------------------------
> This is the analog-help mailing list. To unsubscribe from this
> mailing list, send mail to [EMAIL PROTECTED]
> with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
> List archived at http://www.mail-archive.com/[email protected]/
> ------------------------------------------------------------------------
>
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
