From: Mathias Walter [mailto:[EMAIL PROTECTED]
How can I prevent crawler and bots according to their user agent?
I've put a robots.txt in webapps/ROOT, but this file is not
read again.
So, to check, the crawlers are not reading your robots.txt and are crawling
your site anyway?
I'd like
Hi,
I've put a robots.txt in webapps/ROOT, but this file is not
read again.
So, to check, the crawlers are not reading your robots.txt
and are crawling your site anyway?
I don't know it exactly. The problem is that the sites are linked from
anywhere. I'm not sure, if a crawler that
Hello Mathias,
Mathias Walter wrote:
I don't know it exactly. The problem is that the sites are linked from
anywhere. I'm not sure, if a crawler that follows the link
http://mydomain:port/servlet/page.jsp, looks for the robots.txt in the ROOT
webapp.
Just last week we've installed a
