R. David Murray added the comment: Lucaz pointed out on IRC that the problem is that the current robotparser is implementing an outdated robots.txt standard. He may work on fixing that.
---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue17403> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
