New submission from Karl Y. Pradene <karl.prad...@gmail.com>:
In robotparser.py On line 222 path = urllib.parse.urlunparse(urllib.parse.urlparse(path)) tranform the entry Disallow: /? in the google.com/robots.txt in : Disallow: / making every can_fetch request return False ---------- components: Library (Lib) messages: 394144 nosy: karl.pradene priority: normal severity: normal status: open title: urllib.robotparser fail on Disallow: /? from google.com type: behavior versions: Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue44209> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
