New submission from Tarun Chinmai Sekar <[email protected]>:
IMO this could be enhanced by adding a sslcontext parameter to read method
a sample change would it could look like
```
def read(self, sslcontext=None):
"""Reads the robots.txt URL and feeds it to the parser."""
try:
if sslcontext:
f = urllib.request.urlopen(self.url, context=sslcontext)
else:
f = urllib.request.urlopen(self.url)
except urllib.error.HTTPError as err:
if err.code in (401, 403):
self.disallow_all = True
elif err.code >= 400 and err.code < 500:
self.allow_all = True
else:
raw = f.read()
self.parse(raw.decode("utf-8").splitlines())
```
Happy to send a PR if this proposal makes sense.
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue43597>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
