Anyone know the function of robots.txt? I have seen attempted access to
it in my apache logs.
--
Ken Moffat
[EMAIL PROTECTED]
___
Linux-users mailing list
[EMAIL PROTECTED]
Unsubscribe/Suspend/Etc - http://www.linux-sxs.org/mailman/listinfo/linux
Ken Moffat wrote:
Anyone know the function of robots.txt? I have seen attempted access to
it in my apache logs.
Ah, never mind, I found a page describing this functionality.
http://www.robotstxt.org/wc/exclusion.html
--
Ken Moffat
[EMAIL PROTECTED
On Sun, 9 Feb 2003, Ken Moffat wrote:
Anyone know the function of robots.txt? I have seen attempted access to
it in my apache logs.
Yes. That is where you can limit the browsing of well-behaved robots.
For instance, I run a website with a database of millions of nearly-
identical dynamic
I have deleted apache from my linux server. will not bother to look back.
Ken Moffat wrote:
Ken Moffat wrote:
Anyone know the function of robots.txt? I have seen attempted access to
it in my apache logs.
Ah, never mind, I found a page describing this functionality.
http
On 02/09/03 10:57, Ken Moffat wrote:
M.W. Chang wrote:
I have deleted apache from my linux server. will not bother to look back.
Why?
Please don't feed the troll.
--
~
L. Friedman [EMAIL
Net Llama! wrote:
On 02/09/03 10:57, Ken Moffat wrote:
M.W. Chang wrote:
I have deleted apache from my linux server. will not bother to look
back.
Why?
Please don't feed the troll.
sorry, you are correct.
--
Ken Moffat
[EMAIL PROTECTED]
On Sun, 09 Feb 2003 10:16:40 -0800 (PST) Kevin O'Gorman [EMAIL PROTECTED]
wrote:
On Sun, 9 Feb 2003, Ken Moffat wrote:
Anyone know the function of robots.txt? I have seen attempted access to
it in my apache logs.
Yes. That is where you can limit the browsing of well-behaved robots
