[issue23004] mock_open() should allow reading binary data
Berker Peksag added the comment: data_as_list = read_data.splitlines(True) is not actually the equivalent of data_as_list = [l + sep for l in read_data.split(sep)] It will change the behavior of the _iterate_read_data helper. See the comment at https://github.com/python/cpython/blob/78d05eb847c6b8fede08ca74bb59210c00e4c599/Lib/unittest/mock.py#L2278 However, in default branch, we can simplify it to yield from read_data.splitlines(keepends=True) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
R. David Murray added the comment: splitlines(keepends=True) is not ever equivalent to splitting by just '\n'. I don't know the details here, but switching to that would certainly be a behavior change. (Especially if the code path also applies to non-binary data!): b'abc\nde\r\nf\x1dg'.splitlines(True) [b'abc\n', b'de\r\n', b'f\x1dg'] 'abc\nde\r\nf\x1dg'.splitlines(True) ['abc\n', 'de\r\n', 'f\x1d', 'g'] -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Berker Peksag added the comment: Thanks for the patch, Aaron(also thanks to Demian for reviews). I've fixed the merge conflict and added more tests. -- resolution: - fixed stage: commit review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Roundup Robot added the comment: New changeset 3d7adf5b3fb3 by Berker Peksag in branch '3.4': Issue #23004: mock_open() now reads binary data correctly when the type of read_data is bytes. https://hg.python.org/cpython/rev/3d7adf5b3fb3 New changeset 526a186de32d by Berker Peksag in branch '3.5': Issue #23004: mock_open() now reads binary data correctly when the type of read_data is bytes. https://hg.python.org/cpython/rev/526a186de32d New changeset ed15f399a292 by Berker Peksag in branch 'default': Issue #23004: mock_open() now reads binary data correctly when the type of read_data is bytes. https://hg.python.org/cpython/rev/ed15f399a292 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Robert Collins added the comment: Post merge review: looks like data_as_list = read_data.splitlines(True) would be a little cleaner. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Changes by Berker Peksag berker.pek...@gmail.com: -- assignee: - berker.peksag ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Changes by Robert Collins robe...@robertcollins.net: -- nosy: +rbcollins versions: +Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Berker Peksag added the comment: LGTM. -- components: +Library (Lib) stage: patch review - commit review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Changes by Demian Brecht demianbre...@gmail.com: -- nosy: -demian.brecht ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Demian Brecht added the comment: A few more comments were left in Rietveld for you, likely hidden by spam filters. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Aaron Hill added the comment: I've fixed the formatting issues. -- Added file: http://bugs.python.org/file37510/mock-open-allow-binary-data-fix-formatting.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Aaron Hill added the comment: I've fixed the issues you pointed out. Is there a better way than uploading a new patch file to make changes? -- Added file: http://bugs.python.org/file37458/mock-open-allow-binary-without-coerce-fixup.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Demian Brecht added the comment: Thanks for the updated patch, looks good to me. If you haven't already read it, the patch workflow is here: https://docs.python.org/devguide/patch.html and is the only workflow currently available. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Changes by Berker Peksag berker.pek...@gmail.com: -- stage: needs patch - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Aaron Hill added the comment: Thanks, I've fixed that. Not sure why I thought decoding and re-encoding would work with any binary data. I've also updated one of the tests to use non-utf8-decodeable binary data, to prevent a future regression. -- Added file: http://bugs.python.org/file37446/mock-open-allow-binary-without-coerce.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Demian Brecht added the comment: Thanks again for the update Aaron, I've left a couple small comments in Rietveld. Other than those, the patch looks good to me. Thanks for the contribution! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Aaron Hill added the comment: I've created a new patch, which addresses the problem. Your example now currently returns [b'foo\n', b'bar\n'] -- Added file: http://bugs.python.org/file37440/mock-open-allow-binary-data-updated.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Demian Brecht added the comment:
Thanks for the update, but this doesn't quite work either as you're assuming
utf-8 (which is what .encode() and .decode() default to). For example, when
using latin-1:
m = mock_open(read_data= b'\xc6')
with patch('__main__.open', m, create=True) :
... with open('abc', 'rb') as f :
... print(f.read())
...
Traceback (most recent call last):
[snip]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc6 in position 0:
unexpected end of data
Additionally, a bytes object may simply be binary data that doesn't adhere to
any specific encoding.
My suggestion is to remove the use of format() altogether as it's really not
doing anything complex and simply append either '\n' or b'\n' depending on the
type of object passed in. That way, you can deal with the type of object passed
in directly without coercion.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23004
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Aaron Hill added the comment: I've created a patch that fixes this, and added an accompanying unit test (which fails without the change). -- keywords: +patch nosy: +Aaron1011 Added file: http://bugs.python.org/file37435/mock-open-allow-binary-data.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Demian Brecht added the comment:
Thanks for the patch Aaron. Unfortunately this doesn't quite fix the issue.
There are two problems with the patch:
If a bytes object is passed into mock_open, I'd expect a bytes object in the
output. In your patch, not only is this not the case (the output is a string
object), but the bytes object is being coerced into its string representation
in the resulting list. For example (simplified from your patch):
data = b'foo\nbar'
newline = b'\n'
['{}\n'.format(l) for l in data.split(newline)]
[b'foo'\n, b'bar'\n]
What I would expect to see in this case is:
[b'foo\n', b'bar\n']
--
nosy: +demian.brecht
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23004
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Demian Brecht added the comment: There are two problems with the patch That was intended to be removed after I changed the wording :P -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
New submission from Jesús Cea Avión:
mock_open(read_data=b'...') gives an error:
Traceback (most recent call last):
File z.py, line 6, in module
print(f.read())
File /usr/local/lib/python3.4/unittest/mock.py, line 896, in __call__
return _mock_self._mock_call(*args, **kwargs)
File /usr/local/lib/python3.4/unittest/mock.py, line 962, in _mock_call
ret_val = effect(*args, **kwargs)
File /usr/local/lib/python3.4/unittest/mock.py, line 2279, in
_read_side_effect
return ''.join(_data)
File /usr/local/lib/python3.4/unittest/mock.py, line 2244, in
_iterate_read_data
data_as_list = ['{}\n'.format(l) for l in read_data.split('\n')]
Easy to reproduce:
from unittest.mock import mock_open, patch
m = mock_open(read_data= b'abc')
with patch('__main__.open', m, create=True) :
with open('abc', 'rb') as f :
print(f.read())
Looks like this bug was introduced as result of issue #17467. I add those
people to the nosy list.
--
keywords: easy
messages: 232271
nosy: a.badger, jcea, michael.foord
priority: normal
severity: normal
status: open
title: mock_open() should allow reading binary data
versions: Python 3.5
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23004
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Changes by Jesús Cea Avión j...@jcea.es: -- dependencies: +Enhancement: give mock_open readline() and readlines() methods ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23004] mock_open() should allow reading binary data
Changes by Berker Peksag berker.pek...@gmail.com: -- nosy: +berker.peksag stage: - needs patch type: - behavior versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23004 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
