AbdealiJK added a comment.
I was trying to reproduce your error and was unable to. This is the output I
got:
$ python3 pwb.py weblinkchecker -family:wikipedia -page:Philosophy -day:0
Retrieving 1 pages from wikipedia:en.
>>> Philosophy <<<
1 pages read
0 pages written
Execution time: 2 seconds
Read operation time: 2 seconds
Script terminated successfully.
Waiting for remaining 49 threads to finish, please wait...
Waiting for remaining 40 threads to finish, please wait...
Waiting for remaining 37 threads to finish, please wait...
Waiting for remaining 35 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 33 threads to finish, please wait...
Waiting for remaining 29 threads to finish, please wait...
Waiting for remaining 19 threads to finish, please wait...
Waiting for remaining 8 threads to finish, please wait...
Waiting for remaining 7 threads to finish, please wait...
Waiting for remaining 7 threads to finish, please wait...
Exception while processing URL
http://machaut.uchicago.edu/?resource=Webster%27s&word=epistemology&use1913=on
in page Philosophy
Exception in thread b'Philosophy -
http://machaut.uchicago.edu/?resource=Webster%27s&word=epistemology&use1913=on':
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connection.py",
line 137, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py",
line 67, in create_connection
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/socket.py", line 732,
in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 559, in urlopen
body=body, headers=headers)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
1083, in request
self._send_request(method, url, body, headers)
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
1128, in _send_request
self.endheaders(body)
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
1079, in endheaders
self._send_output(message_body)
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
911, in _send_output
self.send(msg)
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
854, in send
self.connect()
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connection.py",
line 162, in connect
conn = self._new_conn()
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connection.py",
line 146, in _new_conn
self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError:
<requests.packages.urllib3.connection.HTTPConnection object at 0xb4fafeac>:
Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/adapters.py",
line 376, in send
timeout=timeout
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 609, in urlopen
_stacktrace=sys.exc_info()[2])
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py",
line 273, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError:
HTTPConnectionPool(host='machaut.uchicago.edu', port=80): Max retries exceeded
with url: /?resource=Webster%27s&word=epistemology&use1913=on (Caused by
NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object
at 0xb4fafeac>: Failed to establish a new connection: [Errno -2] Name or
service not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/threading.py", line
914, in _bootstrap_inner
self.run()
File "./scripts/weblinkchecker.py", line 568, in run
r = requests.get(self.url, headers=header, timeout=timeout)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/api.py",
line 67, in get
return request('get', url, params=params, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/api.py",
line 53, in request
return session.request(method=method, url=url, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 468, in request
resp = self.send(prep, **send_kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 576, in send
r = adapter.send(request, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/adapters.py",
line 437, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError:
HTTPConnectionPool(host='machaut.uchicago.edu', port=80): Max retries exceeded
with url: /?resource=Webster%27s&word=epistemology&use1913=on (Caused by
NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object
at 0xb4fafeac>: Failed to establish a new connection: [Errno -2] Name or
service not known',))
Waiting for remaining 5 threads to finish, please wait...
Exception while processing URL
http://www.oed.com.ezproxy.uky.edu/view/Entry/142505?rskey=uk0M8u&result=1 in
page Philosophy
Exception in thread b'Philosophy -
http://www.oed.com.ezproxy.uky.edu/view/Entry/142505?rskey=uk0M8u&result=1':
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 559, in urlopen
body=body, headers=headers)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 345, in _make_request
self._validate_conn(conn)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 784, in _validate_conn
conn.connect()
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connection.py",
line 252, in connect
ssl_version=resolved_ssl_version)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/util/ssl_.py",
line 305, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 376, in
wrap_socket
_context=self)
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 747, in
__init__
self.do_handshake()
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 983, in
do_handshake
self._sslobj.do_handshake()
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 628, in
do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/adapters.py",
line 376, in send
timeout=timeout
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 588, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/threading.py", line
914, in _bootstrap_inner
self.run()
File "./scripts/weblinkchecker.py", line 568, in run
r = requests.get(self.url, headers=header, timeout=timeout)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/api.py",
line 67, in get
return request('get', url, params=params, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/api.py",
line 53, in request
return session.request(method=method, url=url, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 468, in request
resp = self.send(prep, **send_kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 597, in send
history = [resp for resp in gen] if allow_redirects else []
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 597, in <listcomp>
history = [resp for resp in gen] if allow_redirects else []
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 195, in resolve_redirects
**adapter_kwargs
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 576, in send
r = adapter.send(request, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/adapters.py",
line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:645)
Waiting for remaining 3 threads to finish, please wait...
Waiting for remaining 3 threads to finish, please wait...
Waiting for remaining 3 threads to finish, please wait...
Waiting for remaining 3 threads to finish, please wait...
Waiting for remaining 1 threads to finish, please wait...
Waiting for remaining 1 threads to finish, please wait...
Waiting for remaining 1 threads to finish, please wait...
Waiting for remaining 1 threads to finish, please wait...
Waiting for remaining 1 threads to finish, please wait...
Exception while processing URL http://www.journals.cambridge.org/phi in
page Philosophy
Exception in thread b'Philosophy - http://www.journals.cambridge.org/phi':
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 378, in _make_request
httplib_response = conn.getresponse()
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
1174, in getresponse
response.begin()
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
282, in begin
version, status, reason = self._read_status()
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line
243, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/socket.py", line 575,
in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/adapters.py",
line 376, in send
timeout=timeout
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 609, in urlopen
_stacktrace=sys.exc_info()[2])
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py",
line 247, in increment
raise six.reraise(type(error), error, _stacktrace)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py",
line 310, in reraise
raise value
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 559, in urlopen
body=body, headers=headers)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 380, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py",
line 308, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" %
timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError:
HTTPConnectionPool(host='www.journals.cambridge.org', port=80): Read timed out.
(read timeout=30)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/threading.py", line
914, in _bootstrap_inner
self.run()
File "./scripts/weblinkchecker.py", line 568, in run
r = requests.get(self.url, headers=header, timeout=timeout)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/api.py",
line 67, in get
return request('get', url, params=params, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/api.py",
line 53, in request
return session.request(method=method, url=url, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 468, in request
resp = self.send(prep, **send_kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/sessions.py",
line 576, in send
r = adapter.send(request, **kwargs)
File
"/home/ajk/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/adapters.py",
line 449, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout:
HTTPConnectionPool(host='www.journals.cambridge.org', port=80): Read timed out.
(read timeout=30)
Saving history...
These aren't issues with python3, but with weblinkchecker.
It only seems to check for `requests.exceptions.InvalidURL` whereas we should
also be checking for `requests.exceptions.SSLError` and
`requests.exceptions.ReadTimeout`
In fact, we should be doing `requests.exceptions.RequestException` - the
error from which all request exceptions inherit.
But that should be another issue in itself.
@jayvdb, do you still get this issue ? If yes, could you elaborate on steps
to reproduce it ?
TASK DETAIL
https://phabricator.wikimedia.org/T124287
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AbdealiJK
Cc: AbdealiJK, Aklapper, StudiesWorld, jayvdb, pywikibot-bugs-list
_______________________________________________
pywikibot-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
