Until the problems with "dumb spiders"[1] are worked around, couldn't something simple be done such as using a spider trap[2]?
Add this to robots.txt: User-agent: * Disallow: /pypi?:action=bot_ban Add a hidden link on the front page, then grab the ip of the offending spider and block it by adding it to .htacess Just a thought, as PyPI has been painfully slow for hours at a time lately. [1] http://mail.python.org/pipermail/catalog-sig/2007-April/001055.html [2] http://danielwebb.us/software/bot-trap/ _______________________________________________ Catalog-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/catalog-sig
