If there is anyone here with an interest in web-spiders, it would be nice if someone else could take a look at
http://bugs.python.org/issue21469
which addresses the risk of false positives with the robots.txt parser.
Raymond
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
