[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: Thanks, Nick! I'll move the function declaration back to abstract.h. Waiting for Georg's input. -- It seems to me that #14330 is a blocker that will only be fixed on Monday. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Georg Brandl ge...@python.org added the comment: With the beta delayed as you say, I'm okay with this going in now. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 8fbbc7c8748e by Stefan Krah in branch 'default': Issue #12834: Fix PyBuffer_ToContiguous() for non-contiguous arrays. http://hg.python.org/cpython/rev/8fbbc7c8748e -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: All right, 3.3 is fixed. Re-targeting for 3.2 and 2.7. -- priority: release blocker - normal versions: +Python 3.1 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: Any objections to committing this before beta2? What about the len view-len change: Does that look reasonable? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Nick Coghlan ncogh...@gmail.com added the comment: Georg, need a call on how close you are to cutting beta 2 and whether you want this to wait until rc1. -- nosy: +benjamin.peterson, georg.brandl priority: normal - release blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Nick Coghlan ncogh...@gmail.com added the comment: Summary of my review for Georg's benefit: I had one minor formatting nit with the patch (which Stefan can easily fix before committing), but it otherwise looked fine to me. I also agree that the old implicit truncation was unusable in practice, as the *actual* length is not returned from the function, and in fact cannot be returned because the return type is int rather than Py_ssize_t. A length mismatch therefore needs to be an error. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: Here's a patch for 3.3, which consists mostly of tests. A couple of points: o I removed the len view-len check in PyBuffer_ToContiguous(), since the function is not documented and silently accepting output buffers that are too large seems unusual to me. o Removed the comment in bytesobject.c Better way to get to internal buffer?. IMO the answer is no: There isn't any better way that works in full generality. o Removed the need to check for overflow comment: ndim is now officially limited to 64. I think this can go into 3.3: It's easy to see that for contiguous buffers PyBuffer_ToContiguous() behaves in the same manner as before (except for the len issue). For memoryview, buffer_to_contiguous(x, y 'C') is literally the same code as buffer_to_c_contiguous(). -- keywords: +patch Added file: http://bugs.python.org/file26512/issue12834.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: There is an additional problem with PyBuffer_ToContiguous(): Suppose 'view' is multi-dimensional, C-contiguous and initialized according to PyBUF_ND, i.e. view-shape != NULL but view-strides == NULL. Now if PyBuffer_ToContiguous() is called with 'F', PyBuffer_IsContiguous() returns false and view-strides will be accessed. This means that incomplete buffer information will have to be reconstructed like it is done in the 3.3 memoryview. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: The fix would require all of these functions from memoryview.c (3.3): last_dim_is_contiguous cmp_structure copy_base copy_rec copy_buffer How to avoid code duplication? I could move them into abstract.c, but conceptually they're really just low level buffer interface functions. Also, they make a lot of assumptions (ndim = 1, PyBUF_FULL) that are a little dangerous for a general interface. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Christian Heimes li...@cheimes.de added the comment: You could move PyBuffer_ToContiguous() from abstract.c to memoryview.c. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: You could move PyBuffer_ToContiguous() from abstract.c to memoryview.c. For 3.3 that would be ideal, yes. I asked a while ago on python-dev whether to backport the memoryview rewrite. The general mood was against it. So, for 2.7/3.2 I could add all these functions to abstract.c. But an additional problem is that the whole test infrastructure of Lib/test/test_buffer.py and Modules/_testbuffer.c would be missing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Christian Heimes li...@cheimes.de added the comment: It looks like Stefan has fixed the issue in Python 3.3 a while ago. tobytes() returns the correct values with a fresh build of Python 3.3. $ PYTHONPATH=. /home/heimes/dev/python/py3k/python smc/freeimage/tests/test_image.py test_newbuffer (__main__.TestImageNewBuffer) ... 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 80 80 80 112 112 112 160 160 160 192 192 192 240 240 240 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 255 255 0 255 0 255 0 255 255 255 255 0 255 0 255 However it's still broken in 3.2 (also up to date hg pull). $ PYTHONPATH=. /home/heimes/dev/python/3.2/python smc/freeimage/tests/test_image.py test_newbuffer (__main__.TestImageNewBuffer) ... 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 80 80 80 112 112 112 160 160 160 192 192 192 240 240 240 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 255 255 0 255 0 255 0 255 255 255 255 0 255 0 255 0 Stefan, could you please port your fix to Python 3.2 and 3.3? Thanks! -- versions: +Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: In Python 3.3 memoryobject.c is a complete rewrite. Porting the fix separately would be quite a bit of work. PyBuffer_ToContiguous(), which causes the problem in 2.7/3.2 and is still broken in 3.3, could be fixed by using the recursive copy_buffer() function from the new memoryobject.c. I don't know if I can fix it before the 3.3 release. When are the next 2.7/3.2 releases? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Christian Heimes li...@cheimes.de added the comment: I think that I run into the same bug today. I've developing a PEP 3118 buffer interface for my wrapper of FreeImage. It returns the data as non-contiguous 3d array with the dimension height, width, color. I've created a small test image with 5x7 RGB pixels. The first line black, second white, third grey values and the rest are red, green blue and cyan, magenta yellow in little endian (BGR order). NumPy handles the buffer correctly: arr = numpy.asarray(img) print(arr) [[[ 0 0 0] [ 0 0 0] [ 0 0 0] [ 0 0 0] [ 0 0 0]] [[255 255 255] [255 255 255] [255 255 255] [255 255 255] [255 255 255]] [[ 80 80 80] [112 112 112] [160 160 160] [192 192 192] [240 240 240]] [[ 0 0 255] [ 0 255 0] [255 0 0] [ 0 0 255] [ 0 255 0]] [[255 0 0] [ 0 0 255] [ 0 255 0] [255 0 0] [ 0 0 255]] [[ 0 255 0] [255 0 0] [ 0 0 255] [ 0 255 0] [255 0 0]] [[255 255 0] [255 0 255] [ 0 255 255] [255 255 0] [255 0 255]]] but memoryview.tobytes() seems to have an off-by-one error: m = memoryview(img) data = m.tobytes() len(data) == 5 * 7 * 3 True for i in range(7): ... print( .join(%3i % ord(v) for v in data[i * 5 * 3:(i + 1) * 5 * 3])) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 80 80 80 112 112 112 160 160 160 192 192 192 240 240 240 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 0 0 255 0 255 0 255 0 0 255 255 0 255 0 255 0 255 255 255 255 0 255 0 255 0 As you can see the first byte is missing. Python 2.7.3 and Python 3.2.3 with numpy 1.6.1 and https://bitbucket.org/tiran/smc.freeimage/changeset/3134ecee2984 on 64bit Ubuntu. -- nosy: +christian.heimes versions: +Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Changes by Meador Inge mead...@gmail.com: -- nosy: +meador.inge ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 3f9b3b6f7ff0 by Stefan Krah in branch 'default': - Issue #10181: New memoryview implementation fixes multiple ownership http://hg.python.org/cpython/rev/3f9b3b6f7ff0 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: I removed the dependency since PyBuffer_ToContiguous() still needs to be fixed in abstract.c while memoryview.tobytes() now works in the PEP-3118 repo. -- dependencies: -Problems with Py_buffer management in memoryobject.c (and elsewhere?) title: memoryview.tobytes() incorrect for non-contiguous arrays - PyBuffer_ToContiguous() incorrect for non-contiguous arrays ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com