[issue22896] Don't use PyObject_As*Buffer() functions

[Martin Panter]

Martin Panter added the comment:

Please see Issue 25498 for a crash possibly caused by the memoryview hack in 
CDataType_from_buffer().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Roundup Robot]

Roundup Robot added the comment:

New changeset 1da9630e9b7f by Serhiy Storchaka in branch '3.4':
Issue #22896: Avoid to use PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
https://hg.python.org/cpython/rev/1da9630e9b7f

New changeset 2e684ce772de by Serhiy Storchaka in branch 'default':
Issue #22896: Avoid to use PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
https://hg.python.org/cpython/rev/2e684ce772de

New changeset 0024537a4687 by Serhiy Storchaka in branch 'default':
Issue #22896: Fixed using _getbuffer() in recently added _PyBytes_Format().
https://hg.python.org/cpython/rev/0024537a4687

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In any case we need a hack in 3.4. Let open new issue for adding 
PyMemoryView_FromObjectEx() or like.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Stefan Krah]

Stefan Krah added the comment:

Thanks.  No, I don't think there's an official way to accomplish that,
but let's create one.  How about a new function that takes the buffer
request flags:

PyMemoryView_FromObjectEx(exporter, PyBUF_SIMPLE|PyBUF_WRITABLE)

If we can spare a new format code, this could be called directly in
PyArg_ParseTuple(), which would give back the memoryview.

Otherwise, you get the exporter from PyArg_ParseTuple() and call
PyMemoryView_FromObjectEx() manually.

If I'm not mistaken, this would save us the intermediate buffer on the
stack, and it's more readable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thanks Antoine and Stefan for your reviews.

--
resolution:  - fixed
stage: patch review - resolved
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Stefan Krah]

Stefan Krah added the comment:

Nice patch.  I've found one issue (see Rietveld).  I'm not sure
about 3.4 (the patch contains minor refactorings), but otherwise
I'd say go ahead with it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you please look at the patch Stefan?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Stefan Krah]

Stefan Krah added the comment:

[Slow internet connection, can't use Rietveld.]

CDataType_from_buffer():

I'm not that familiar with ctypes.  What is the high level goal here?
Allocate a chunk of memory, wrap it in a memoryview and have the
memoryview release that memory when its refcount is 0?

If so, I think we desperately need direct support for that in
memoryview.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

from_buffer() uses a memory buffer of other object. It keeps a reference to the 
object to prevent deallocation of memory when there will be no more external 
references. But this doesn't prevent from reallocating of memory of living 
object (e.g. for resizing of bytearray). So we need to grab the buffer (with 
PyObject_GetBuffer) in from_buffer() and free it (with PyBuffer_Release) when 
it is no longer needed. menoryview can do this but needs a hack, because a 
memoryview created by PyMemoryView_FromBuffer() doesn't release the buffer. May 
be there is more official way?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Martin Panter]

Martin Panter added the comment:

_CData.from_buffer() is meant to take a writable buffer, and create a “ctypes” 
object that shares the same memory. So it should not release the buffer until 
that “ctypes” object is no longer needed.

However I don’t know the insides of memoryview() objects that well so I can’t 
say if the hack-the-memoryview code is correct or not.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Oh, Martin expressed the same thing better.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is simplified patch. _PyBuffer_Converter() and simple_buffer converter 
are gone. Fixed few leaks found by Martin. Fixed potential crash in ctypes. 
Fixed minor bugs and cleaned up ctypes tests for from_buffer().

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


Added file: http://bugs.python.org/file37514/buffers_3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Updated patch addresses Antoine's comments.

I still hesitate about C-contiguousity. Looks as all buffers created in the 
stdlib are C-contiguous, so we can't test non-contiguous buffers. Shouldn't 
PyObject_AsCharBuffer (or even PyObject_AsReadBuffer and_PyBuffer_Converter) 
accept only C-contiguous buffers? Who are buffer protocol experts?

--
Added file: http://bugs.python.org/file37507/buffers_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Shouldn't PyObject_AsCharBuffer (or even PyObject_AsReadBuffer 
 and_PyBuffer_Converter) accept only C-contiguous buffers?

PyBUF_SIMPLE enforces contiguity. See 
https://www.python.org/dev/peps/pep-3118/#access-flags and 
https://docs.python.org/3/c-api/buffer.html#c.Py_buffer.len

Also Stefan's post at 
http://mail.scipy.org/pipermail/numpy-discussion/2011-August/058189.html

Perhaps Stefan can confirm.

--
nosy: +skrah

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ah, there is a way to create non-contiguous buffers.

 b = bytes(range(16))
 m = memoryview(b)
 m[::2].c_contiguous
False

 PyBUF_SIMPLE enforces contiguity.

Then contiguousity check in getbuffer() in Python/getargs.c is redundant. And 
in most cases the use of _PyBuffer_Converter() and PyObject_GetBuffer() could 
be replaced by the use of y* and w* format units.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Stefan Krah]

Stefan Krah added the comment:

Yes, a PyBUF_SIMPLE request implies c-contiguous, so it's ok.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
keywords: +needs review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Alexander Belopolsky]

Changes by Alexander Belopolsky alexander.belopol...@gmail.com:


--
nosy: +belopolsky

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is a patch which replaces deprecated functions with PyObject_GetBuffer() 
and like. It also introduce _PyBuffer_Converter for using in PyArgs_Parse* and 
clinic converter simple_buffer_converter which unlike to Py_buffer_converter 
(y*) does not not force C contiguousity (is it correct?).

--
keywords: +patch
nosy: +larry, pitrou
stage: needs patch - patch review
Added file: http://bugs.python.org/file37272/buffers.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22896] Don't use PyObject_As*Buffer() functions

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

PyObject_AsCharBuffer(), PyObject_AsReadBuffer() and PyObject_AsWriteBuffer() 
release the buffer right after acquiring and return a pointer to released 
buffer. This is not safe and could cause issues sooner or later. These 
functions shouldn't be used in the stdlib at all.

--
assignee: serhiy.storchaka
components: Extension Modules, Interpreter Core
messages: 231323
nosy: serhiy.storchaka
priority: normal
severity: normal
stage: needs patch
status: open
title: Don't use PyObject_As*Buffer() functions
type: resource usage
versions: Python 3.4, Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22896
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com