Richard Hansen added the comment: (The following message is mostly off-topic but I think it is relevant to those interested in this issue. This message is about the clarity of the documentation regarding flag semantics, and what I think the flags should mean.)
> Cython doesn't follow the spec though (use Python 3): > > from _testbuffer import * > cpdef foo(): > cdef unsigned char[:] v = bytearray(b"testing") > nd = ndarray(v, getbuf=PyBUF_ND) > print(nd.suboffsets) > nd = ndarray(v, getbuf=PyBUF_FULL) > print(nd.suboffsets) When I compile and run the above (latest Cython from Git master), I get: () () Looking at the Python source code (Modules/_testbuffer.c ndarray_get_suboffsets() and ssize_array_as_tuple()) the above output is printed if the suboffsets member is NULL. > Cython hands out suboffsets even for a PyBUF_ND request, which is > definitely wrong. For a PyBUF_FULL request they should only be > provided *if needed*. suboffsets appears to be NULL in both cases in your example, which seems acceptable to me given: * the user didn't request suboffsets information in the PyBUF_ND case, and * there are no indirections in the PyBUF_FULL case. Even if suboffsets wasn't NULL, I believe the behavior would still be correct for the PyBUF_ND case. My reading of <https://docs.python.org/3/c-api/buffer.html> is that flags=PyBUF_ND implies that shape member MUST NOT be NULL, but strides and suboffsets can be NULL or not (it doesn't matter). This interpretation of mine is due to the sentence, "Note that each flag contains all bits of the flags below it." If flags=PyBUF_ND meant that strides and suboffsets MUST be NULL, then PyBUF_ND and PyBUF_STRIDES would necessarily be mutually exclusive and flags=(PyBUF_ND|PyBUF_STRIDES) would not make sense (the strides member would have to be both NULL and non-NULL). If (flags & PyBUF_INDIRECT) is false, then the consumer is not interested in the suboffsets member so it shouldn't matter what it points to. (And the consumer should not dereference the pointer in case it points to a junk address.) IMHO, if the buffer is C-style contiguous then the producer should be allowed to populate the shape, strides, and suboffsets members regardless of whether or not any of the PyBUF_ND, PyBUF_STRIDES, or PyBUF_INDIRECT flags are set. In other words, for C-style contiguous buffers, the producer should be allowed to act as if PyBUF_INDIRECT was always provided because the consumer will always get an appropriate Py_buffer struct regardless of the actual state of the PyBUF_ND, PyBUF_STRIDES, and PyBUF_INDIRECT flags. It *is* a bug, however, to dereference the strides or suboffsets members with ndarray(v, getbuf=PyBUF_ND) because the consumer didn't ask for strides or suboffsets information and the pointers might be bogus. Stepping back a bit, I believe that the flags should be thought of as imposing requirements on the producer. I think those requirements should be (assuming ndim > 0): * PyBUF_ND: If (flags & PyBUF_ND) is true: - If (flags & PyBUF_STRIDES) is false *and* the producer is unable to produce a block of memory at [buf,buf+len) containing all (len/itemsize) entries in a C-style contiguous chunk, then the producer must raise an exception. - Otherwise, the producer must provide the shape of buffer. If (flags & PyBUF_ND) is false: - If the producer is unable to produce a contiguous chunk of (len/itemsize) entries (of any shape) in the block of memory at [buf,buf+len), then the producer must raise an exception. - Otherwise, the producer is permitted to do any of the following: + don't touch the shape member (don't set it to NULL or any other value; just leave it as-is) + set the shape member to NULL + set the shape member to point to an array describing the shape + set the shape member to point to any other location, even if dereferencing the pointer would result in a segfault * PyBUF_STRIDES: If (flags & PyBUF_STRIDES) is true: - The producer must provide the appropriate strides information. If (flags & PyBUF_STRIDES) is false: - If the producer is unable to produce a block of memory at [buf,buf+len) containing all (len/itemsize) entries, the producer must raise an exception. - Otherwise, the producer is permitted to do any of the following; + don't touch the strides member (don't set it to NULL or any other value; just leave it as-is) + set the strides member to NULL + set the strides member to point to an array describing the strides + set the strides member to point to any other location, even if dereferencing the pointer would result in a segfault * PyBUF_INDIRECT: If (flags & PyBUF_INDIRECT) is true: - If the buffer uses indirections then the producer must set the suboffsets member to point to an array with appropriate entries. - Otherwise, the producer can either set the suboffsets member to NULL or set it to point to an array of all negative entries. If (flags & PyBUF_INDIRECT) is false: - If the producer cannot produce a buffer that does not have indirections, then the producer must raise an exception. - Otherwise, the producer is permitted to do any of the following: + don't touch the suboffsets member (don't set it to NULL or any other value; just leave it as-is) + set the suboffsets member to NULL + set the suboffsets member to point to an array of all negative entries + set the suboffsets member to point to any other location, even if dereferencing the pointer would result in a segfault ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue23352> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com