[Cython] buffer shape incompatible with memoryview shape
Hi, I find this worth fixing for 0.17: http://trac.cython.org/cython_trac/ticket/780 Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
On 21 June 2012 09:59, Stefan Behnel wrote: > Hi, > > I find this worth fixing for 0.17: > > http://trac.cython.org/cython_trac/ticket/780 > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel It seems that arrays are compared as pointer values, so it doesn't even compare sensibly anyway. You can easily work around it by writing ( memoryview).shape though. I think these shape/strides/suboffset arrays should have a special type and coerce to tuples when coercing to an object. Feel free to work on that, it wouldn't really require touching much or any of the memoryview code, it's not really on m priority list right now. BTW, Stefan, how do we start Jenkins on the sage server? It's been down for weeks now. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Jenkins status
mark florisson, 21.06.2012 12:38: > BTW, Stefan, how do we start Jenkins on the sage server? It's been > down for weeks now. It seems like the sage.math server would be happy about a restart. I'll trigger the ML. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
On 06/21/2012 10:59 AM, Stefan Behnel wrote: Hi, I find this worth fixing for 0.17: http://trac.cython.org/cython_trac/ticket/780 I'm not sure about the timeline here. The object<->memoryview semantics haven't even been hammered down yet; does "mview.customattr" trigger an AttributeError, SyntaxError or fall back to some underlying object (constructing it if necesarry). Until that happens, memoryviews are an experimental feature and present for development purposes mostly, so it's not like this is a big bug that would bite end-users. Thinking about those semantics is much more important... Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
Dag Sverre Seljebotn, 21.06.2012 13:10: > On 06/21/2012 10:59 AM, Stefan Behnel wrote: >> I find this worth fixing for 0.17: >> >> http://trac.cython.org/cython_trac/ticket/780 > > I'm not sure about the timeline here. > > The object<->memoryview semantics haven't even been hammered down yet; does > "mview.customattr" trigger an AttributeError, SyntaxError or fall back to > some underlying object (constructing it if necesarry). > > Until that happens, memoryviews are an experimental feature and present for > development purposes mostly, so it's not like this is a big bug that would > bite end-users. Thinking about those semantics is much more important... Absolutely. I ran into this when I gave a Cython+NumPy course and this was the first thing that the attendants tried when I asked them to validate that two input arrays have the same size before adding them. It's the one obvious way to do it, and it fails miserably. I think it should be fixed, and I think it should be fixed soon because it feels really low-level and complicated, especially to new users. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
On 06/21/2012 01:36 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 21.06.2012 13:10: On 06/21/2012 10:59 AM, Stefan Behnel wrote: I find this worth fixing for 0.17: http://trac.cython.org/cython_trac/ticket/780 I'm not sure about the timeline here. The object<->memoryview semantics haven't even been hammered down yet; does "mview.customattr" trigger an AttributeError, SyntaxError or fall back to some underlying object (constructing it if necesarry). Until that happens, memoryviews are an experimental feature and present for development purposes mostly, so it's not like this is a big bug that would bite end-users. Thinking about those semantics is much more important... Absolutely. I ran into this when I gave a Cython+NumPy course and this was the first thing that the attendants tried when I asked them to validate that two input arrays have the same size before adding them. It's the one obvious way to do it, and it fails miserably. I think it should be fixed, and I think it should be fixed soon because it feels really low-level and complicated, especially to new users. Can you clarify a bit -- did you give this course using np.ndarray[double, ndim=2], or double[:, :]? They're really very separate under the hood and the fix is different. Or, did you actually use object[double, ndim=2] like in the bug report? (Did me and Mark get around to propose deprecating this one on the list?) Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
Dag Sverre Seljebotn, 21.06.2012 14:05: > On 06/21/2012 01:36 PM, Stefan Behnel wrote: >>> On 06/21/2012 10:59 AM, Stefan Behnel wrote: I find this worth fixing for 0.17: http://trac.cython.org/cython_trac/ticket/780 >>> >> I ran into this when I gave a Cython+NumPy course and this was the first >> thing that the attendants tried when I asked them to validate that two >> input arrays have the same size before adding them. It's the one obvious >> way to do it, and it fails miserably. I think it should be fixed, and I >> think it should be fixed soon because it feels really low-level and >> complicated, especially to new users. > > Can you clarify a bit -- did you give this course using np.ndarray[double, > ndim=2], or double[:, :]? They're really very separate under the hood and > the fix is different. > > Or, did you actually use object[double, ndim=2] like in the bug report? > (Did me and Mark get around to propose deprecating this one on the list?) IIRC, we used object[double, ndim=2] for both and I also tried it with a memory view as in the bug report. I thought that using "object" was the preferred way to do it? At least, it doesn't restrict the type of the buffer exporter, which I consider a good thing. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
On 06/21/2012 02:59 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 21.06.2012 14:05: On 06/21/2012 01:36 PM, Stefan Behnel wrote: On 06/21/2012 10:59 AM, Stefan Behnel wrote: I find this worth fixing for 0.17: http://trac.cython.org/cython_trac/ticket/780 I ran into this when I gave a Cython+NumPy course and this was the first thing that the attendants tried when I asked them to validate that two input arrays have the same size before adding them. It's the one obvious way to do it, and it fails miserably. I think it should be fixed, and I think it should be fixed soon because it feels really low-level and complicated, especially to new users. Can you clarify a bit -- did you give this course using np.ndarray[double, ndim=2], or double[:, :]? They're really very separate under the hood and the fix is different. Or, did you actually use object[double, ndim=2] like in the bug report? (Did me and Mark get around to propose deprecating this one on the list?) IIRC, we used object[double, ndim=2] for both and I also tried it with a memory view as in the bug report. I thought that using "object" was the preferred way to do it? At least, it doesn't restrict the type of the buffer exporter, which I consider a good thing. That's a very theoretical argument as NumPy arrays are in practice the only exporter. I always teach np.ndarray[double...]. I've never told anyone about object[...], I don't think it's in much use. For starters it's going to be horribly inefficient unless you also add "mode='strided'" within the brackets. My proposal (and Mark's I think) is: Since the memoryviews will neatly cover the general exporter case, and since the [] syntax is much overloaded already (used for C++ templates too), we should deprecate object[...] no matter what else happens. Depending on what's decided for np.ndarray[...], we have: Case A): Deprecate both np.ndarray[...] and object[...] Case B): Only deprecate object[...], keep np.ndarray[...] (e.g., through a decorator used in numpy.pxd on the ndarray type). So rather than having a trailing [] mean buffers unless it means something else (like C++ templates), we instead make np.ndarray a "template", through special compiler support. Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
Dag Sverre Seljebotn, 21.06.2012 15:06: > On 06/21/2012 02:59 PM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 21.06.2012 14:05: >>> On 06/21/2012 01:36 PM, Stefan Behnel wrote: > On 06/21/2012 10:59 AM, Stefan Behnel wrote: >> I find this worth fixing for 0.17: >> >> http://trac.cython.org/cython_trac/ticket/780 > I ran into this when I gave a Cython+NumPy course and this was the first thing that the attendants tried when I asked them to validate that two input arrays have the same size before adding them. It's the one obvious way to do it, and it fails miserably. I think it should be fixed, and I think it should be fixed soon because it feels really low-level and complicated, especially to new users. >>> >>> Can you clarify a bit -- did you give this course using np.ndarray[double, >>> ndim=2], or double[:, :]? They're really very separate under the hood and >>> the fix is different. >>> >>> Or, did you actually use object[double, ndim=2] like in the bug report? >>> (Did me and Mark get around to propose deprecating this one on the list?) >> >> IIRC, we used object[double, ndim=2] for both and I also tried it with a >> memory view as in the bug report. I thought that using "object" was the >> preferred way to do it? At least, it doesn't restrict the type of the >> buffer exporter, which I consider a good thing. > > That's a very theoretical argument as NumPy arrays are in practice the only > exporter. Except for, say, bytes objects, array.array and user implemented types, that is. lxml has buffer support for its serialised XSLT output, for example. > I always teach np.ndarray[double...]. I've never told anyone about > object[...], I don't think it's in much use. For starters it's going to be > horribly inefficient unless you also add "mode='strided'" within the brackets. Ah, good to know. > My proposal (and Mark's I think) is: > > Since the memoryviews will neatly cover the general exporter case, and > since the [] syntax is much overloaded already (used for C++ templates > too), we should deprecate object[...] no matter what else happens. > > Depending on what's decided for np.ndarray[...], we have: > > Case A): Deprecate both np.ndarray[...] and object[...] > > Case B): Only deprecate object[...], keep np.ndarray[...] (e.g., through a > decorator used in numpy.pxd on the ndarray type). So rather than having a > trailing [] mean buffers unless it means something else (like C++ > templates), we instead make np.ndarray a "template", through special > compiler support. What's the point in technically deprecating them if we can't remove them without breaking code? Wouldn't it be better to deprecate them only in the docs? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
On 21 June 2012 14:34, Stefan Behnel wrote: > Dag Sverre Seljebotn, 21.06.2012 15:06: >> On 06/21/2012 02:59 PM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn, 21.06.2012 14:05: On 06/21/2012 01:36 PM, Stefan Behnel wrote: >> On 06/21/2012 10:59 AM, Stefan Behnel wrote: >>> I find this worth fixing for 0.17: >>> >>> http://trac.cython.org/cython_trac/ticket/780 >> > I ran into this when I gave a Cython+NumPy course and this was the first > thing that the attendants tried when I asked them to validate that two > input arrays have the same size before adding them. It's the one obvious > way to do it, and it fails miserably. I think it should be fixed, and I > think it should be fixed soon because it feels really low-level and > complicated, especially to new users. Can you clarify a bit -- did you give this course using np.ndarray[double, ndim=2], or double[:, :]? They're really very separate under the hood and the fix is different. Or, did you actually use object[double, ndim=2] like in the bug report? (Did me and Mark get around to propose deprecating this one on the list?) >>> >>> IIRC, we used object[double, ndim=2] for both and I also tried it with a >>> memory view as in the bug report. I thought that using "object" was the >>> preferred way to do it? At least, it doesn't restrict the type of the >>> buffer exporter, which I consider a good thing. >> >> That's a very theoretical argument as NumPy arrays are in practice the only >> exporter. > > Except for, say, bytes objects, array.array and user implemented types, > that is. lxml has buffer support for its serialised XSLT output, for example. > You can already easily obtain a pointer from a bytes object, which is already 1d anyways :) Whether buffers on array.array are useful is still questionable given their variably-sized nature. >> I always teach np.ndarray[double...]. I've never told anyone about >> object[...], I don't think it's in much use. For starters it's going to be >> horribly inefficient unless you also add "mode='strided'" within the >> brackets. > > Ah, good to know. > > >> My proposal (and Mark's I think) is: >> >> Since the memoryviews will neatly cover the general exporter case, and >> since the [] syntax is much overloaded already (used for C++ templates >> too), we should deprecate object[...] no matter what else happens. > I agree with deprecating the object[] syntax, I think memoryviews should prove themselves a bit more for e.g. 0.17, before we start deprecating np.ndarray. >> Depending on what's decided for np.ndarray[...], we have: >> >> Case A): Deprecate both np.ndarray[...] and object[...] >> >> Case B): Only deprecate object[...], keep np.ndarray[...] (e.g., through a >> decorator used in numpy.pxd on the ndarray type). So rather than having a >> trailing [] mean buffers unless it means something else (like C++ >> templates), we instead make np.ndarray a "template", through special >> compiler support. > > What's the point in technically deprecating them if we can't remove them > without breaking code? Wouldn't it be better to deprecate them only in the > docs? > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer shape incompatible with memoryview shape
On 21 June 2012 13:05, Dag Sverre Seljebotn wrote: > On 06/21/2012 01:36 PM, Stefan Behnel wrote: >> >> Dag Sverre Seljebotn, 21.06.2012 13:10: >>> >>> On 06/21/2012 10:59 AM, Stefan Behnel wrote: I find this worth fixing for 0.17: http://trac.cython.org/cython_trac/ticket/780 >>> >>> >>> I'm not sure about the timeline here. >>> >>> The object<->memoryview semantics haven't even been hammered down yet; >>> does >>> "mview.customattr" trigger an AttributeError, SyntaxError or fall back to >>> some underlying object (constructing it if necesarry). >>> >>> Until that happens, memoryviews are an experimental feature and present >>> for >>> development purposes mostly, so it's not like this is a big bug that >>> would >>> bite end-users. Thinking about those semantics is much more important... >> >> >> Absolutely. >> >> I ran into this when I gave a Cython+NumPy course and this was the first >> thing that the attendants tried when I asked them to validate that two >> input arrays have the same size before adding them. It's the one obvious >> way to do it, and it fails miserably. I think it should be fixed, and I >> think it should be fixed soon because it feels really low-level and >> complicated, especially to new users. > > > Can you clarify a bit -- did you give this course using np.ndarray[double, > ndim=2], or double[:, :]? They're really very separate under the hood and > the fix is different. I think we should support both, although it seems a bit of a shame to fix something just a while before deprecating it :) Anyway, both fixes are really straightforward anyway. > Or, did you actually use object[double, ndim=2] like in the bug report? (Did > me and Mark get around to propose deprecating this one on the list?) > > Dag > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
