[issue20699] Behavior of ZipFile with file-like object and BufferedWriter.
Martin Panter added the comment: Posting patch v2: * Changed readinto() argument descriptions to “a pre-allocated, writable bytes-like buffer”, for both RawIOBase and BufferedIOBase * Integrated the single-use test_memoryio.BytesIOMixin test class, which tricked me when I did the first patch * Added tests for BufferedRWPair, BytesIO.readinto() etc methods with non-bytearray() buffers * Fix _pyio.BufferedReader.readinto/1() for non-bytearray -- Added file: http://bugs.python.org/file38107/bytes-like-param.v2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20699] Behavior of ZipFile with file-like object and BufferedWriter.
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20699] Behavior of ZipFile with file-like object and BufferedWriter.
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka stage: - patch review versions: +Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20699] Behavior of ZipFile with file-like object and BufferedWriter.
Martin Panter added the comment: I think the simplest thing to do here would be to update the documentation to match the usage. This patch does so, saying that all write() methods, as well as the BytesIO() constructor, have to accept bytes-like objects. It also expands some tests to verify this, and fixes a resulting bug in _pyio. -- assignee: - docs@python components: +Documentation keywords: +patch nosy: +docs@python Added file: http://bugs.python.org/file37662/bytes-like-param.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20699] Behavior of ZipFile with file-like object and BufferedWriter.
Changes by Ned Deily n...@acm.org: -- nosy: +benjamin.peterson, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20699] Behavior of ZipFile with file-like object and BufferedWriter.
Martin Panter added the comment: I have a related issue in Python 3.4. I suspect it is the same underlying problem as Henning’s. BufferedWriter is trying to write memoryview() objects, but the documentation for RawIOBase.write() implies it only has to accept bytes() and bytearray() objects. from io import BufferedWriter, RawIOBase class Raw(RawIOBase): ... def writable(self): return True ... def write(self, b): print(b.startswith(b\n)) ... b = BufferedWriter(Raw()) b.write(babc) 3 b.close() Traceback (most recent call last): File stdin, line 1, in module File stdin, line 3, in write AttributeError: 'memoryview' object has no attribute 'startswith' -- nosy: +vadmium versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20699] Behavior of ZipFile with file-like object and BufferedWriter.
New submission from Henning von Bargen: Regression: Behavior of ZipFile with file-like object and BufferedWriter. The following code worked with Python 2.6: LOB_BLOCKSIZE = 1024*1024 # 1 MB class UnbufferedBlobWriter(io.RawIOBase): A file-like wrapper for a write-only cx_Oracle BLOB object. def __init__(self, blobLocator): self.blobLocator = blobLocator self.offset = 0 self.blobLocator.open() def seekable(self): return True def seek(self, offset, whence): if whence == 0: self.offset = offset elif whence == 1: self.offset += offset if self.offset 0: self.offset = 0 elif whence == 2: if offset = 0 and -offset = self.blobLocator.size(): self.offset = self.blobLocator.size() + offset else: raise IOError(96, Invalid offset for BlobWriter) else: self._unsupported(seek) return self.offset def writable(self): return True def write(self, data, offset=None): if offset is None: offset = self.offset self.blobLocator.write(bytes(data), offset + 1) self.offset = offset + len(data) return len(data) def close(self): self.flush() self.blobLocator.close() def BlobWriter(blobLocator): A file-like wrapper (buffered) for a write-only cx_Oracle BLOB object. return io.BufferedWriter(UnbufferedBlobWriter(blobLocator), LOB_BLOCKGROESSE) Note: The cx_Oracle BLOB object is used to store binary content inside a database. It's basically a file-like-like object. I'm using it in conjunction with a ZipFile object to store a ZipFile as a BLOB inside the DB, like this: curs.execute( insert into ... values (..., Empty_BLOB()) returning BDATA into :po_BDATA , [..., blobvar]) blob = BlobWriter(blobvar.getvalue()) archive = ZipFile(blob, w, ZIP_DEFLATED) for filename in ...: self.log.debug(Saving to ZIP file in the DB: %s, filename) archive.write(filename, filename) archive.close() This used to work with Python 2.6. With Python 2.7.5 however, somethin like this gets written into the blob: memory at 0x.. Digging deeper, I found out that when using the UnbufferedBlobWriter directly (without BufferedWriter), the problem does not occur. It seems like the behaviour of the BufferedWriter class changed from 2.6 to 2.7, most probably caused by the internal optimization of using the memoryview class. As a workaround, I had to change my write method, calling tobytes() if necessary: def write(self, data, offset=None): if offset is None: offset = self.offset if hasattr(data, tobytes): self.blobLocator.write(data.tobytes(), offset + 1) else: self.blobLocator.write(bytes(data), offset + 1) self.offset = offset + len(data) return len(data) I'm not sure if this is actually a bug in 2.7 or if my usage of BufferedWriter is incorrect (see remark). For understanding the problem it is important to know that the ZipFile.write method often calls write and seek. Remark: If I am mis-using BufferedWriter: What precisely is wrong? And if so, why is it so complicated to support a buffered-random-writer? I cannot use io.BufferedRandom because I don't have a read method (and ZipFile.write does not need that). -- components: IO messages: 211714 nosy: Henning.von.Bargen priority: normal severity: normal status: open title: Behavior of ZipFile with file-like object and BufferedWriter. type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
