Richard Hansen added the comment:

(The following message is mostly off-topic but I think it is relevant to those 
interested in this issue.  This message is about the clarity of the 
documentation regarding flag semantics, and what I think the flags should mean.)

> Cython doesn't follow the spec though (use Python 3):
> 
>     from _testbuffer import *
>     cpdef foo():
>         cdef unsigned char[:] v = bytearray(b"testing")
>         nd = ndarray(v, getbuf=PyBUF_ND)
>         print(nd.suboffsets)
>         nd = ndarray(v, getbuf=PyBUF_FULL)
>         print(nd.suboffsets)

When I compile and run the above (latest Cython from Git master), I get:

    ()
    ()

Looking at the Python source code (Modules/_testbuffer.c 
ndarray_get_suboffsets() and ssize_array_as_tuple()) the above output is 
printed if the suboffsets member is NULL.

> Cython hands out suboffsets even for a PyBUF_ND request, which is
> definitely wrong.  For a PyBUF_FULL request they should only be
> provided *if needed*.

suboffsets appears to be NULL in both cases in your example, which seems 
acceptable to me given:
  * the user didn't request suboffsets information in the PyBUF_ND
    case, and
  * there are no indirections in the PyBUF_FULL case.

Even if suboffsets wasn't NULL, I believe the behavior would still be correct 
for the PyBUF_ND case.  My reading of 
<https://docs.python.org/3/c-api/buffer.html> is that flags=PyBUF_ND implies 
that shape member MUST NOT be NULL, but strides and suboffsets can be NULL or 
not (it doesn't matter).  This interpretation of mine is due to the sentence, 
"Note that each flag contains all bits of the flags below it."  If 
flags=PyBUF_ND meant that strides and suboffsets MUST be NULL, then PyBUF_ND 
and PyBUF_STRIDES would necessarily be mutually exclusive and 
flags=(PyBUF_ND|PyBUF_STRIDES) would not make sense (the strides member would 
have to be both NULL and non-NULL).

If (flags & PyBUF_INDIRECT) is false, then the consumer is not interested in 
the suboffsets member so it shouldn't matter what it points to.  (And the 
consumer should not dereference the pointer in case it points to a junk 
address.)

IMHO, if the buffer is C-style contiguous then the producer should be allowed 
to populate the shape, strides, and suboffsets members regardless of whether or 
not any of the PyBUF_ND, PyBUF_STRIDES, or PyBUF_INDIRECT flags are set.  In 
other words, for C-style contiguous buffers, the producer should be allowed to 
act as if PyBUF_INDIRECT was always provided because the consumer will always 
get an appropriate Py_buffer struct regardless of the actual state of the 
PyBUF_ND, PyBUF_STRIDES, and PyBUF_INDIRECT flags.

It *is* a bug, however, to dereference the strides or suboffsets members with 
ndarray(v, getbuf=PyBUF_ND) because the consumer didn't ask for strides or 
suboffsets information and the pointers might be bogus.

Stepping back a bit, I believe that the flags should be thought of as imposing 
requirements on the producer.  I think those requirements should be (assuming 
ndim > 0):

  * PyBUF_ND:  If (flags & PyBUF_ND) is true:
      - If (flags & PyBUF_STRIDES) is false *and* the producer is
        unable to produce a block of memory at [buf,buf+len)
        containing all (len/itemsize) entries in a C-style contiguous
        chunk, then the producer must raise an exception.
      - Otherwise, the producer must provide the shape of buffer.
    If (flags & PyBUF_ND) is false:
      - If the producer is unable to produce a contiguous chunk of
        (len/itemsize) entries (of any shape) in the block of memory
        at [buf,buf+len), then the producer must raise an exception.
      - Otherwise, the producer is permitted to do any of the
        following:
          + don't touch the shape member (don't set it to NULL or any
            other value; just leave it as-is)
          + set the shape member to NULL
          + set the shape member to point to an array describing the
            shape
          + set the shape member to point to any other location, even
            if dereferencing the pointer would result in a segfault
  * PyBUF_STRIDES:  If (flags & PyBUF_STRIDES) is true:
      - The producer must provide the appropriate strides information.
    If (flags & PyBUF_STRIDES) is false:
      - If the producer is unable to produce a block of memory at
        [buf,buf+len) containing all (len/itemsize) entries, the
        producer must raise an exception.
      - Otherwise, the producer is permitted to do any of the
        following;
          + don't touch the strides member (don't set it to NULL or
            any other value; just leave it as-is)
          + set the strides member to NULL
          + set the strides member to point to an array describing the
            strides
          + set the strides member to point to any other location,
            even if dereferencing the pointer would result in a
            segfault
  * PyBUF_INDIRECT:  If (flags & PyBUF_INDIRECT) is true:
      - If the buffer uses indirections then the producer must set the
        suboffsets member to point to an array with appropriate
        entries.
      - Otherwise, the producer can either set the suboffsets member
        to NULL or set it to point to an array of all negative
        entries.
    If (flags & PyBUF_INDIRECT) is false:
      - If the producer cannot produce a buffer that does not have
        indirections, then the producer must raise an exception.
      - Otherwise, the producer is permitted to do any of the
        following:
          + don't touch the suboffsets member (don't set it to NULL or
            any other value; just leave it as-is)
          + set the suboffsets member to NULL
          + set the suboffsets member to point to an array of all
            negative entries
          + set the suboffsets member to point to any other location,
            even if dereferencing the pointer would result in a
            segfault

----------

_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue23352>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Reply via email to