On Apr 2, 2008, at 11:11 , Allen Schmidt Sr. wrote:
Thanks Chris.
Seeing tons of squid messages about the new msnbot....and it is not following our robots.txt directive. In fact, I have even disallowed it completely and it still goes where it wants. And the updated version was supposed to have been released end of last week....about the time our zeo clients started pegging 99%.

Trusting that bots actually care about or respect robots.txt won't get you very far. If all else fails I suggest you look at your logs to find out what IPs or IP ranges those bots come from and block them at the firewall level, using e.g. iptables if you're on Linux. This is a very simplistic invocation:

iptables -A INPUT -s <source-IP> -j DROP

At least that way you gain some breathing room to come up with a less constrictive solution.


Zope maillist  -  Zope@zope.org
**   No cross posts or HTML encoding!  **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

Reply via email to