Re: [Numpy-discussion] numpy.int32, type inheritance and tp_flags
On Mon, Jun 1, 2009 at 11:58 PM, David Cournapeau < da...@ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Mon, Jun 1, 2009 at 11:08 PM, David Cournapeau > > mailto:da...@ar.media.kyoto-u.ac.jp>> > > wrote: > > > > Hi, > > > >I have a question related to #1121 > > (http://projects.scipy.org/numpy/ticket/1121). With python 2.6, > > PyInt_Check(a) if a is an instance of numpy.int32 does not work > > anymore. > > It think this is related to the python issue 2263 > > > > > > (http://bugs.python.org/issue2263), where the tp_flags has been > > changed > > for the python int object, change which influences PyInt_Check > > behavior. > > > > > > It would be nice if the python folks would document > > Py_TPFLAGS_INT_SUBCLASS so we knew what it did. I also wonder if the > > problem with struct and the related bug with timeseries aren't python > > bugs. Shouldn't python be checking for conversion calls rather than an > > integer subclass? > > I found this while walking through the python hg log: > > http://www.mail-archive.com/python-...@python.org/msg18140.html > Hmm, makes me think even more that the python code is the buggy one here. Why should it depend on inheritance from the int type? I mean, isn't that kind of limiting? All they need to know is that it can be *converted* to a python integer. It's like duck typing is being replaced by strict typing. Because it's faster to interpret (duh). Of course, I may have no idea what I'm talking about. > > As I understand it, that's basically an optimization for fast subclass > testing, and is indeed not documented. But instead of hard-coding the > additional flag for types which support this in numpy, I think it would > be better to have something which will not break again when another flag > is added to some types. There speaks the build guy ;) > Specially since related bugs are quite hard to > track. I don't know how to do it, though, as the python doc says that > inheriting tp_flags is tricky... > Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.int32, type inheritance and tp_flags
Charles R Harris wrote: > > > On Mon, Jun 1, 2009 at 11:08 PM, David Cournapeau > mailto:da...@ar.media.kyoto-u.ac.jp>> > wrote: > > Hi, > >I have a question related to #1121 > (http://projects.scipy.org/numpy/ticket/1121). With python 2.6, > PyInt_Check(a) if a is an instance of numpy.int32 does not work > anymore. > It think this is related to the python issue 2263 > > > (http://bugs.python.org/issue2263), where the tp_flags has been > changed > for the python int object, change which influences PyInt_Check > behavior. > > > It would be nice if the python folks would document > Py_TPFLAGS_INT_SUBCLASS so we knew what it did. I also wonder if the > problem with struct and the related bug with timeseries aren't python > bugs. Shouldn't python be checking for conversion calls rather than an > integer subclass? I found this while walking through the python hg log: http://www.mail-archive.com/python-...@python.org/msg18140.html As I understand it, that's basically an optimization for fast subclass testing, and is indeed not documented. But instead of hard-coding the additional flag for types which support this in numpy, I think it would be better to have something which will not break again when another flag is added to some types. Specially since related bugs are quite hard to track. I don't know how to do it, though, as the python doc says that inheriting tp_flags is tricky... David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.int32, type inheritance and tp_flags
On Mon, Jun 1, 2009 at 11:50 PM, David Cournapeau < da...@ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Mon, Jun 1, 2009 at 11:08 PM, David Cournapeau > > mailto:da...@ar.media.kyoto-u.ac.jp>> > > wrote: > > > > Hi, > > > >I have a question related to #1121 > > (http://projects.scipy.org/numpy/ticket/1121). With python 2.6, > > PyInt_Check(a) if a is an instance of numpy.int32 does not work > > anymore. > > It think this is related to the python issue 2263 > > (http://bugs.python.org/issue2263), where the tp_flags has been > > changed > > for the python int object, change which influences PyInt_Check > > behavior. > > > > What should we do about it ? Right now, it looks like the > > bitfields are > > harcoded in scalar types - shouldn't we inherit them from the > original > > python types (in a field per field manner) instead ? > > > > > > Maybe, but why should it work for int32 anyway? > > Because it does at the Python level ? > > issubclass(np.int32, int) # True > > And some code depends on this (I noticed the problem while tracking down > some issues for scipy 0.7.x on python 2.6), although the code could be > modified to not depend on it anymore I guess. > > > IIRC, the python int type has different lengths on windows and linux > > 64 bit systems. > > Yes, because the underlying C type is a long (at least for python 2.5.4 > as I read it from Include/intobject.h in the sources). Windows (with MS > compilers at least) reserves 4 bytes only for long on 64 bits. > > But is numpy.int32 really a subclass of int on 64 bits ? I played a bit > with numpy on python 2.4 64 bits (Linux): > No, IIRC, int64 is. You can see this in the different behavior, i.e., it doesn't act like the other numpy scalars. > > import numpy as np > int(2**33) # Returns the right value, of type 'int' > np.int32(2**33) # Oups, 0 > > On 32 bits: > > import numpy as np > int(2*33) # Returns the right value, of type 'long' > np.int32(2**33) # 66 ... > > And what about 3.0? > > There is not python 2.* int anymore, only python 2.* long object (which > becomes the sole int object on py3k). The PyInt_* apis are gone too, > starting from 3.1. > Exactly, and that's going to hurt. Macro time ;) > > > > > I think we probably need to do something here, but I'm not sure what. > > The different behavior of the numpy double and integer types > > corresponding to the python types as opposed to the rest of the scalar > > types is an issue that has annoyed me since forever. > > I think for now I will just add a workaround in scipy. That's probably the safest thing at the moment. > I don't > understand much about scalar types, so I don't have a clue about what to > do - I feel that it will be one dark area for 3.* porting, though :) > Me too, on all counts. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.int32, type inheritance and tp_flags
On Mon, Jun 1, 2009 at 11:08 PM, David Cournapeau < da...@ar.media.kyoto-u.ac.jp> wrote: > Hi, > >I have a question related to #1121 > (http://projects.scipy.org/numpy/ticket/1121). With python 2.6, > PyInt_Check(a) if a is an instance of numpy.int32 does not work anymore. > It think this is related to the python issue 2263 > (http://bugs.python.org/issue2263), where the tp_flags has been changed > for the python int object, change which influences PyInt_Check behavior. > It would be nice if the python folks would document Py_TPFLAGS_INT_SUBCLASS so we knew what it did. I also wonder if the problem with struct and the related bug with timeseries aren't python bugs. Shouldn't python be checking for conversion calls rather than an integer subclass? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.int32, type inheritance and tp_flags
Charles R Harris wrote: > > > On Mon, Jun 1, 2009 at 11:08 PM, David Cournapeau > mailto:da...@ar.media.kyoto-u.ac.jp>> > wrote: > > Hi, > >I have a question related to #1121 > (http://projects.scipy.org/numpy/ticket/1121). With python 2.6, > PyInt_Check(a) if a is an instance of numpy.int32 does not work > anymore. > It think this is related to the python issue 2263 > (http://bugs.python.org/issue2263), where the tp_flags has been > changed > for the python int object, change which influences PyInt_Check > behavior. > > What should we do about it ? Right now, it looks like the > bitfields are > harcoded in scalar types - shouldn't we inherit them from the original > python types (in a field per field manner) instead ? > > > Maybe, but why should it work for int32 anyway? Because it does at the Python level ? issubclass(np.int32, int) # True And some code depends on this (I noticed the problem while tracking down some issues for scipy 0.7.x on python 2.6), although the code could be modified to not depend on it anymore I guess. > IIRC, the python int type has different lengths on windows and linux > 64 bit systems. Yes, because the underlying C type is a long (at least for python 2.5.4 as I read it from Include/intobject.h in the sources). Windows (with MS compilers at least) reserves 4 bytes only for long on 64 bits. But is numpy.int32 really a subclass of int on 64 bits ? I played a bit with numpy on python 2.4 64 bits (Linux): import numpy as np int(2**33) # Returns the right value, of type 'int' np.int32(2**33) # Oups, 0 On 32 bits: import numpy as np int(2*33) # Returns the right value, of type 'long' np.int32(2**33) # 66 ... > And what about 3.0? There is not python 2.* int anymore, only python 2.* long object (which becomes the sole int object on py3k). The PyInt_* apis are gone too, starting from 3.1. > > I think we probably need to do something here, but I'm not sure what. > The different behavior of the numpy double and integer types > corresponding to the python types as opposed to the rest of the scalar > types is an issue that has annoyed me since forever. I think for now I will just add a workaround in scipy. I don't understand much about scalar types, so I don't have a clue about what to do - I feel that it will be one dark area for 3.* porting, though :) cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.int32, type inheritance and tp_flags
On Mon, Jun 1, 2009 at 11:08 PM, David Cournapeau < da...@ar.media.kyoto-u.ac.jp> wrote: > Hi, > >I have a question related to #1121 > (http://projects.scipy.org/numpy/ticket/1121). With python 2.6, > PyInt_Check(a) if a is an instance of numpy.int32 does not work anymore. > It think this is related to the python issue 2263 > (http://bugs.python.org/issue2263), where the tp_flags has been changed > for the python int object, change which influences PyInt_Check behavior. > > What should we do about it ? Right now, it looks like the bitfields are > harcoded in scalar types - shouldn't we inherit them from the original > python types (in a field per field manner) instead ? > Maybe, but why should it work for int32 anyway? IIRC, the python int type has different lengths on windows and linux 64 bit systems. And what about 3.0? I think we probably need to do something here, but I'm not sure what. The different behavior of the numpy double and integer types corresponding to the python types as opposed to the rest of the scalar types is an issue that has annoyed me since forever. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.int32, type inheritance and tp_flags
Hi, I have a question related to #1121 (http://projects.scipy.org/numpy/ticket/1121). With python 2.6, PyInt_Check(a) if a is an instance of numpy.int32 does not work anymore. It think this is related to the python issue 2263 (http://bugs.python.org/issue2263), where the tp_flags has been changed for the python int object, change which influences PyInt_Check behavior. What should we do about it ? Right now, it looks like the bitfields are harcoded in scalar types - shouldn't we inherit them from the original python types (in a field per field manner) instead ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
