On Thu, 29 Apr 1999, Johan Friberg wrote:
> Andre Assaiante ([EMAIL PROTECTED]) wrote:
>
> >Folks:
> >
> >Assuming that all well-behaved spiders request /robots.txt at some point in
> >their session, is there a way to not only remove the single request for this
> >file (FILEEXCLUDE /robots.txt), but remove the whole session or at least
> >remove the host for the day?
> >
>
> Since I�m a Newbie on web* and analog I noticed first today in my log
> that there is a
> /robots.txt. What does it mean ?
>
See http://info.webcrawler.com/mak/projects/robots/norobots.html
--
Stephen Turner [EMAIL PROTECTED] http://www.statslab.cam.ac.uk/~sret1/
Statistical Laboratory, 16 Mill Lane, Cambridge CB2 1SB, England
--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------
