[issue15903] Make rawiobase_read() read directly to bytes object

Change by Mark Lawrence : -- nosy: -BreamoreBoy ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue15903] Make rawiobase_read() read directly to bytes object

Change by Inada Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue15903] Make rawiobase_read() read directly to bytes object

Changes by Stefan Krah stefan-use...@bytereef.org: -- nosy: -skrah ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15903 ___ ___ Python-bugs-list

[issue15903] Make rawiobase_read() read directly to bytes object

Mark Lawrence added the comment: Please note this is also referred to from #15994. -- nosy: +BreamoreBoy versions: +Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15903 ___

[issue15903] Make rawiobase_read() read directly to bytes object

Dwight Guth added the comment: I was programming something today and thought I should let you know I came across a situation where the current behavior of this function is able to expose what seems to be raw memory to the user. import io class A(io.RawIOBase): def readinto(self, b):

[issue15903] Make rawiobase_read() read directly to bytes object

Stefan Krah added the comment: So the problem is that readinto(view) might result in several references to view? I don't think that can be solved on the memoryview side. One could do: view = PyMemoryView_FromObject(b); // Lie about writability ((PyMemoryViewObject

[issue15903] Make rawiobase_read() read directly to bytes object

Richard Oudkerk added the comment: Then the view owns a reference to the bytes object. But that does not solve the problem that writable memoryviews based on a readonly object might be hanging around. How about doing PyObject_GetBuffer(b, buf, PyBUF_WRITABLE); view =

[issue15903] Make rawiobase_read() read directly to bytes object

Richard Oudkerk added the comment: The current non-test uses of PyMemoryView_FromBuffer() are in _io.BufferedReader.read(), _io.BufferedWriter.write(), PyUnicode_Decode(). It looks like they can each be made to leak a memoryview that references a deallocated buffer. (Maybe the answer is

[issue15903] Make rawiobase_read() read directly to bytes object

Richard Oudkerk added the comment: I am rather confused about the ownership semantics when one uses PyMemoryView_FromBuffer(). It looks as though PyMemoryView_FromBuffer() steals ownership of the buffer since, when the associated _PyManagedBufferObject is garbage collected,

[issue15903] Make rawiobase_read() read directly to bytes object

Stefan Krah added the comment: Richard Oudkerk rep...@bugs.python.org wrote: PyObject_GetBuffer(b, buf, PyBUF_WRITABLE); view = PyMemoryView_FromBuffer(buf); // readinto view PyBuffer_Release(buf); Would attempts to access a leaked reference to view now result in

[issue15903] Make rawiobase_read() read directly to bytes object

Stefan Krah added the comment: Richard Oudkerk rep...@bugs.python.org wrote: The documentation is not very helpful. It just says that calls to PyObject_GetBuffer() must be matched with calls to PyBuffer_Release(). Yes, we need to sort that out, see #15821. --

[issue15903] Make rawiobase_read() read directly to bytes object

Richard Oudkerk added the comment: You would need to call memory_release(). Perhaps we can just expose it on the C-API level as PyMemoryView_Release(). Should PyMemoryView_Release() release the _PyManagedBufferObject by doing mbuf_release(view-mbuf) even if view-mbuf-exports 0? Doing

[issue15903] Make rawiobase_read() read directly to bytes object

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15903 ___ ___ Python-bugs-list

[issue15903] Make rawiobase_read() read directly to bytes object

Antoine Pitrou added the comment: So I think the combination of PyMemoryView_FromObject() with a call to PyMemoryView_Release() should indeed work here. I don't think we want to expose a mutable bytes object to outside code, so IMO PyMemoryView_FromMemory() is preferrable. --

[issue15903] Make rawiobase_read() read directly to bytes object

Stefan Krah added the comment: Richard Oudkerk rep...@bugs.python.org wrote: Should PyMemoryView_Release() release the _PyManagedBufferObject by doing mbuf_release(view-mbuf) even if view-mbuf-exports 0? No, I think it should really just be a wrapper: diff --git a/Objects/memoryobject.c

[issue15903] Make rawiobase_read() read directly to bytes object

Stefan Krah added the comment: Antoine Pitrou rep...@bugs.python.org wrote: I don't think we want to expose a mutable bytes object to outside code, so IMO PyMemoryView_FromMemory() is preferrable. I agree, but PyMemoryView_FromMemory(PyBytes_AS_STRING(b), n, PyBUF_WRITE) just hides the fact

[issue15903] Make rawiobase_read() read directly to bytes object

Antoine Pitrou added the comment: Antoine Pitrou rep...@bugs.python.org wrote: I don't think we want to expose a mutable bytes object to outside code, so IMO PyMemoryView_FromMemory() is preferrable. I agree, but PyMemoryView_FromMemory(PyBytes_AS_STRING(b), n, PyBUF_WRITE) just hides

[issue15903] Make rawiobase_read() read directly to bytes object

Richard Oudkerk added the comment: Are we talking about a big speedup here or could we perhaps just keep the existing code? I doubt it is worth the hassle. But I did want to know if there was a clean way to do what I wanted. -- ___ Python

[issue15903] Make rawiobase_read() read directly to bytes object

Richard Oudkerk added the comment: New patch which checks the refcount of the memoryview and bytes object after calling readinto(). If either refcount is larger than the expected value of 1, then the data is copied rather than resized. -- Added file:

[issue15903] Make rawiobase_read() read directly to bytes object

Changes by Serhiy Storchaka storch...@gmail.com: -- components: +IO, Library (Lib) type: - performance ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15903 ___

[issue15903] Make rawiobase_read() read directly to bytes object

Antoine Pitrou added the comment: If either refcount is larger than the expected value of 1, then the data is copied rather than resized. I think that's a useless precaution. The bytes object cannot leak since you are using PyMemoryView_FromMemory(), which doesn't know about the original

[issue15903] Make rawiobase_read() read directly to bytes object

Richard Oudkerk added the comment: I think that's a useless precaution. The bytes object cannot leak since you are using PyMemoryView_FromMemory(), which doesn't know about the original object. The bytes object cannot leak so, as you say, checking that refcount is pointless. But the view

[issue15903] Make rawiobase_read() read directly to bytes object

Antoine Pitrou added the comment: The bytes object cannot leak so, as you say, checking that refcount is pointless. But the view might leak, and since it does not own a reference to the base object we have a problem: we can't deallocate the bytes object for fear of breaking the view.

[issue15903] Make rawiobase_read() read directly to bytes object

New submission from Richard Oudkerk: Currently rawiobase_read() reads to a bytearray object and then copies the data to a bytes object. There is a TODO comment saying that the bytes object should be created directly. The attached patch does that. -- files: iobase_read.patch

[issue15903] Make rawiobase_read() read directly to bytes object

Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea stage: - patch review versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15903 ___

[issue15903] Make rawiobase_read() read directly to bytes object

Amaury Forgeot d'Arc added the comment: It works as along as the bytes object cannot leak to Python code, (imagine a custom readinto() method which plays with gc.get_referrers, then calls hash(b)...) This is OK with this patch. -- nosy: +amaury.forgeotdarc