| rafidaslam created this task. rafidaslam added a project: Pywikibot-core. Herald added subscribers: pywikibot-bugs-list, Aklapper. |
TASK DESCRIPTION
The stream=True argument doesn't effect in pywikibot.comms.http.fetch(), it doesn't jump immediately to another statement like requests.get() (with stream=True) does.
(pywikibot-core) refeed@rhTeK:~/W/core:master$ python pwb.py shell
Welcome to the Pywikibot interactive shell!
>>> pywikibot.comms.http
<module 'pywikibot.comms.http' from '/home/rafid/WikimediaGerrit/core/pywikibot/comms/http.py'>
>>> pywikibot.comms.http.fetch
<function fetch at 0x7fdedde88ae8>
>>> def test_fetch():
... resp = pywikibot.comms.http.fetch('https://dumps.wikimedia.org/idwiki/latest/idwiki-latest-abstract.xml', stream=True)
... for chunk in resp.data.iter_content(1024):
... print('test...............')
...
>>> test_fetch()
^CTraceback (most recent call last):
File "<console>", line 1, in <module>
File "<console>", line 2, in test_fetch
File "/home/rafid/WikimediaGerrit/core/pywikibot/comms/http.py", line 506, in fetch
request = _enqueue(uri, method, params, body, headers, **kwargs)
File "/home/rafid/WikimediaGerrit/core/pywikibot/comms/http.py", line 463, in _enqueue
_http_process(session, request)
File "/home/rafid/WikimediaGerrit/core/pywikibot/comms/http.py", line 377, in _http_process
verify=not ignore_validation)
File "/home/rafid/.venvs/pywikibot-core/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/home/rafid/.venvs/pywikibot-core/lib/python3.6/site-packages/requests/sessions.py", line 658, in send
r.content
File "/home/rafid/.venvs/pywikibot-core/lib/python3.6/site-packages/requests/models.py", line 823, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/home/rafid/.venvs/pywikibot-core/lib/python3.6/site-packages/requests/models.py", line 745, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/home/rafid/.venvs/pywikibot-core/lib/python3.6/site-packages/urllib3/response.py", line 436, in stream
data = "" decode_content=decode_content)
File "/home/rafid/.venvs/pywikibot-core/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
data = ""
File "/usr/lib64/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/usr/lib64/python3.6/http/client.py", line 493, in readinto
n = self.fp.readinto(b)
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 1009, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 871, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
KeyboardInterrupt
# Since it takes a long time, and it downloads a big file (over 1G) I canceled it.
# However, with requests.get()
>>> import requests
>>>
>>> def test_requests_get():
... resp = requests.get('https://dumps.wikimedia.org/idwiki/latest/idwiki-latest-abstract.xml', stream=True)
... for chunk in resp.iter_content(1024):
... print('test...................')
...
>>> test_requests_get()
test...................
test...................
test...................
test...................
test...................
test...................
test...................
test...................
test...................
test...................
test...................
test...................TASK DETAIL
EMAIL PREFERENCES
To: rafidaslam
Cc: Aklapper, pywikibot-bugs-list, rafidaslam, Magul, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, jayvdb, Dalba, Masti, Alchimista, Rxy
Cc: Aklapper, pywikibot-bugs-list, rafidaslam, Magul, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, jayvdb, Dalba, Masti, Alchimista, Rxy
_______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
