jenkins-bot has submitted this change. (
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1352475?usp=email )
Change subject: Avoid reading streamed HTTP responses to detect encoding
......................................................................
Avoid reading streamed HTTP responses to detect encoding
Keep streamed response bodies unread until callers consume them. Retain
the encoding chosen by requests unless an explicit charset is provided,
and preserve existing encoding detection for non-streamed responses.
Change-Id: Ia0394aec9f1ca5c58f9d8f9934c1dfb0686fb5d1
---
M pywikibot/comms/http.py
M tests/http_tests.py
2 files changed, 34 insertions(+), 3 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py
index bf09359..d6ef262 100644
--- a/pywikibot/comms/http.py
+++ b/pywikibot/comms/http.py
@@ -438,6 +438,8 @@
The *body* parameter was removed; use *data* instead.
.. version-changed:: 11.8
Avoid mutating headers inputs.
+ .. version-changed:: 11.9
+ Streamed responses no longer read the body to detect its encoding.
See :py:obj:`requests.Session.request` for parameters.
@@ -450,9 +452,12 @@
overridden by domain in config.
:keyword charset: Either a valid charset (usable for str.decode())
- or None to automatically chose the charset from the returned
- header (defaults to latin-1)
+ or None to automatically choose the charset from the returned
+ header (defaults to latin-1 when not streaming)
:type charset: CodecInfo, str, None
+ :keyword bool stream: Defer reading the response body until consumed.
+ When True, use the explicit *charset* or retain the encoding chosen
+ by requests from the response headers, without inspecting the body.
:keyword verify: Verify the SSL certificate (default is True)
:type verify: bool or path to certificates
:keyword callbacks: Methods to call once data is fetched
@@ -526,7 +531,11 @@
except Exception as e:
response = e
else:
- response.encoding = _decide_encoding(response, charset)
+ if kwargs.get('stream'):
+ if charset is not None:
+ response.encoding = charset
+ else:
+ response.encoding = _decide_encoding(response, charset)
for callback in callbacks:
# Note: error_handling_callback raises the Exception
diff --git a/tests/http_tests.py b/tests/http_tests.py
index 7f8284b..3a9d926 100755
--- a/tests/http_tests.py
+++ b/tests/http_tests.py
@@ -12,6 +12,7 @@
import warnings
from contextlib import suppress
from http import HTTPStatus
+from io import BytesIO
from platform import python_implementation
from unittest.mock import Mock, patch
@@ -251,6 +252,27 @@
net = False
+ def test_streaming_does_not_read_body(self) -> None:
+ """Leave streamed bodies unread until the caller consumes them."""
+ content = b'<?xml version="1.0" encoding="UTF-8"?><test />'
+ for charset in (None, 'latin1'):
+ with self.subTest(charset=charset):
+ response = requests.Response()
+ response.status_code = HTTPStatus.OK
+ response.headers['content-type'] = 'application/xml'
+ response.request = requests.Request()
+ response.encoding = 'utf-8'
+ response.raw = BytesIO(content)
+
+ with patch.object(http.session, 'request',
+ return_value=response):
+ with http.fetch('https://example.test', stream=True,
+ charset=charset) as result:
+ self.assertEqual(response.raw.tell(), 0)
+ self.assertEqual(result.encoding, charset or 'utf-8')
+ self.assertEqual(b''.join(result.iter_content(8)),
+ content)
+
def test_inputs_are_not_modified(self) -> None:
"""Test that headers and callbacks remain unchanged."""
headers = {'X-Caller': 'value'}
--
To view, visit
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1352475?usp=email
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings?usp=email
Gerrit-MessageType: merged
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Ia0394aec9f1ca5c58f9d8f9934c1dfb0686fb5d1
Gerrit-Change-Number: 1352475
Gerrit-PatchSet: 1
Gerrit-Owner: Mahveotm <[email protected]>
Gerrit-Reviewer: Xqt <[email protected]>
Gerrit-Reviewer: jenkins-bot
_______________________________________________
Pywikibot-commits mailing list -- [email protected]
To unsubscribe send an email to [email protected]
