On Apr 2, 2008, at 11:11 , Allen Schmidt Sr. wrote:
Thanks Chris.
Seeing tons of squid messages about the new msnbot....and it is not
following our robots.txt directive. In fact, I have even disallowed
it completely and it still goes where it wants. And the updated
version was supposed to have been released end of last week....about
the time our zeo clients started pegging 99%.
Trusting that bots actually care about or respect robots.txt won't get
you very far. If all else fails I suggest you look at your logs to
find out what IPs or IP ranges those bots come from and block them at
the firewall level, using e.g. iptables if you're on Linux. This is a
very simplistic invocation:
iptables -A INPUT -s <source-IP> -j DROP
At least that way you gain some breathing room to come up with a less
constrictive solution.
jens
_______________________________________________
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
