Jens Vagelpohl wrote:

Trusting that bots actually care about or respect robots.txt won't get you very far. If all else fails I suggest you look at your logs to find out what IPs or IP ranges those bots come from and block them at the firewall level, using e.g. iptables if you're on Linux. This is a very simplistic invocation:

iptables -A INPUT -s <source-IP> -j DROP

At least that way you gain some breathing room to come up with a less constrictive solution.

Also very effective against errant Google Search Appliances in intranet situtations ;-)


Simplistix - Content Management, Zope & Python Consulting
Zope maillist  -
**   No cross posts or HTML encoding!  **
(Related lists - )

Reply via email to