On Sun, 09 Feb 2003 10:16:40 -0800 (PST) Kevin O'Gorman <[EMAIL PROTECTED]>
wrote:
> On Sun, 9 Feb 2003, Ken Moffat wrote:
>
> > Anyone know the function of robots.txt? I have seen attempted access to
> > it in my apache logs.
> >
>
> Yes. That is where you can limit the browsing of well-behaved robots.
It's also used to control other things besides web-crawlers... for instance wget
honors it.
--
******************************************************************************
Registered Linux User Number 185956
http://groups.google.com/groups?hl=en&safe=off&group=linux
Join me in chat at #linux-users on irc.freenode.net
5:41pm up 32 days, 23:15, 2 users, load average: 0.15, 0.13, 0.09
_______________________________________________
Linux-users mailing list
[EMAIL PROTECTED]
Unsubscribe/Suspend/Etc -> http://www.linux-sxs.org/mailman/listinfo/linux-users
