Re: [Numpy-discussion] NumPy SVN broken
2009/10/8 Charles R Harris charlesr.har...@gmail.com: code generator problem and doesn't call for a jump in major version. We hope ;) I think David's hack, which looks to have been committed by Stefan, should fix things up. I accidentally committed some of David's patches, but I reverted them back out. I think David's idea of generating an API from dictionary is much cleaner. We can work on implementing that today. Cheers Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy SVN broken
Stéfan van der Walt wrote: We can work on implementing that today. I am working on it ATM - it is taking me longer than expected, though. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy SVN broken
On Oct 7, 2009, at 9:51 PM, David Cournapeau wrote: On Thu, Oct 8, 2009 at 11:39 AM, Travis Oliphant oliph...@enthought.com wrote: I apologize for the mis communication that has occurred here. No problem I did not understand that there was a desire to keep ABI compatibility with NumPy 1.3 when NumPy 1.4 was released.The datetime merge was made under that presumption. I had assumed that people would be fine with recompilation of extension modules that depend on the NumPy C-API.There are several things that needed to be done to merge in new fundamental data-types. Why don't we call the next release NumPy 2.0 if that helps things? Personally, I'd prefer that over hacks to keep ABI compatibility. Keeping ABI compatibility by itself is not an hack - the current workaround is an hack, but that's only because the current way of doing things in code generator is a bit ugly, and I did not want to spend too much time on it. It is purely an implementation issue, the fundamental idea is straightforward. If you want a cleaner solution, I can work on it. I think the hour or so that it would take is worth it compared to breaking many people's code. If that's all it would take, then definitely go for it.I'm not sure breaking people's code is the right image, though. It's more like forcing people to upgrade to take advantage of new features. Improvements to the encapsulation of the numpy C-API are definitely welcome. They have come a long way from their beginnings in Numeric already due to the efforts of you and David Cooke (and I'm sure others I'm not as aware of). The problem I have with spending time on it though is that there is still more implementation work to finish on the datetime functionality to complete the NEP implementation. Naturally, I'd like to see those improvements made first. But, time-spent is usually a function of how much time it takes to get-in to the code, so I won't try to distract you if you have a clear idea about how to proceed. -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] byteswapping a complex scalar
On Oct 7, 2009, at 10:28 AM, Michael Droettboom wrote:
I'm noticing an inconsistency as to how complex numbers are
byteswapped
as arrays vs. scalars, and wondering if I'm doing something wrong.
x = np.array([-1j], 'c8')
x.tostring().encode('hex')
'80bf'
# This is a little-endian representation, in the order (real, imag)
# When I swap the whole array, it swaps each of the (real, imag) parts
separately
y = x.byteswap()
y.tostring().encode('hex')
'bf80'
# and this round-trips fine
z = np.fromstring(y.tostring(), dtype='c8')
assert z[0] == -1j
# When I swap the scalar, it seems to swap the entire 8 bytes
y = x[0].byteswap()
y.tostring().encode('hex')
'bf80'
# ...and this doesn't round-trip
z = np.fromstring(y.tostring(), dtype='c8')
assert z[0] == -1j
Traceback (most recent call last):
File stdin, line 1, in module
AssertionError
Any thoughts?
I think this is a bug. You should file a ticket and mark it
critical.As I look at the scalar implementation (in
gentype_byteswap in scalartypes.c.src), it looks like it's basing it
just on the size (Hmm I don't know why it's not using the copyswap
in the descr field). This works for many types, but not complex
numbers which should have real and imaginary parts handled separately.
There are two ways to fix this that I can see:
1) fix the gentype implementation to use the copyswap function
pointer from the datatype object
2) over-ride the byteswap in the complex scalar Python type
(there is a base-class complex scalar type where it could be placed)
to do the right thing.
I would probably do #1 if I get a chance to work on it (because
strings shouldn't be byteswapped either and they currently are, I
see...)
x = np.array(['abcd'])
Compare:
x.byteswap()[0]
x[0].byteswap()
The work around is to byteswap before extraction:
x.byteswap()[0]
Thanks for the bug-report.
-Travis
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] byteswapping a complex scalar
On Oct 7, 2009, at 10:28 AM, Michael Droettboom wrote:
I'm noticing an inconsistency as to how complex numbers are
byteswapped
as arrays vs. scalars, and wondering if I'm doing something wrong.
x = np.array([-1j], 'c8')
x.tostring().encode('hex')
'80bf'
# This is a little-endian representation, in the order (real, imag)
# When I swap the whole array, it swaps each of the (real, imag) parts
separately
y = x.byteswap()
y.tostring().encode('hex')
'bf80'
# and this round-trips fine
z = np.fromstring(y.tostring(), dtype='c8')
assert z[0] == -1j
# When I swap the scalar, it seems to swap the entire 8 bytes
y = x[0].byteswap()
y.tostring().encode('hex')
'bf80'
# ...and this doesn't round-trip
z = np.fromstring(y.tostring(), dtype='c8')
assert z[0] == -1j
Traceback (most recent call last):
File stdin, line 1, in module
AssertionError
Any thoughts?
I just checked a fix for this into SVN (tests still need to be added
though...)
I can't currently build SVN on my Mac for some reason (I don't know if
it has to do with recent changes or not, but I don't have time to
track it down right nowthe error I'm getting is something about
Datetime array scalar types not being defined which seems related to
the work Dave and Stefan have been discussing).
It's a small change, though, and should work.
-Travis
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] byteswapping a complex scalar
On Oct 8, 2009, at 7:19 AM, Travis Oliphant wrote: I just checked a fix for this into SVN (tests still need to be added though...) I can't currently build SVN on my Mac for some reason (I don't know if it has to do with recent changes or not, but I don't have time to track it down right nowthe error I'm getting is something about Datetime array scalar types not being defined which seems related to the work Dave and Stefan have been discussing). I can build from SVN. The problem is I had to check-out again from SVN (and get rid of the old code-generated files --- sure would be nice if there were the equivalent of make clean -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy SVN broken
On Thu, Oct 8, 2009 at 8:55 PM, Travis Oliphant oliph...@enthought.com wrote: On Oct 7, 2009, at 9:51 PM, David Cournapeau wrote: On Thu, Oct 8, 2009 at 11:39 AM, Travis Oliphant oliph...@enthought.com wrote: I apologize for the mis communication that has occurred here. No problem I did not understand that there was a desire to keep ABI compatibility with NumPy 1.3 when NumPy 1.4 was released. The datetime merge was made under that presumption. I had assumed that people would be fine with recompilation of extension modules that depend on the NumPy C-API. There are several things that needed to be done to merge in new fundamental data-types. Why don't we call the next release NumPy 2.0 if that helps things? Personally, I'd prefer that over hacks to keep ABI compatibility. Keeping ABI compatibility by itself is not an hack - the current workaround is an hack, but that's only because the current way of doing things in code generator is a bit ugly, and I did not want to spend too much time on it. It is purely an implementation issue, the fundamental idea is straightforward. If you want a cleaner solution, I can work on it. I think the hour or so that it would take is worth it compared to breaking many people's code. If that's all it would take, then definitely go for it. I'm not sure breaking people's code is the right image, though. It's more like forcing people to upgrade to take advantage of new features. We got several people complaining about segfaults and the like - granted, those could have been avoided by updating the ABI accordingly. The problem I have with spending time on it though is that there is still more implementation work to finish on the datetime functionality to complete the NEP implementation. Naturally, I'd like to see those improvements made first. But, time-spent is usually a function of how much time it takes to get-in to the code, so I won't try to distract you if you have a clear idea about how to proceed. I am applying my changes as we speak - it took me much more time than I wished because I tried hard to make sure the ABI was not changed. But at least, the current scheme should be much more robust: the ordering is fixed at one single place, and there are a few checks which ensure we don't screw things up (by putting 'holes' in the api array, or by using twice the same index). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] byteswapping a complex scalar
On Oct 8, 2009, at 12:08 PM, Michael Droettboom wrote: Thanks! I guess I won't file a bug then ;) Probably still should, actually: Until the tests get committed, the bug is not really fixed -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] PyArray_FROM_OF from Cython
I'm trying to use PyArray_FROM_OF from Cython and the generated C code
keeps crashing. Dag said on the Cython list that he wasn't sure what
was going on, so maybe someone here will have an idea.
The line that gdb says is crashing is:
#0 0x00e48287 in __pyx_pf_3_vq_vq (__pyx_self=0x0,
__pyx_args=0xca2d8, __pyx_kwds=0x0) at _vq_rewrite.c:1025
1025 __pyx_t_1 = PyArray_FROM_OF(((PyObject *)__pyx_v_obs),
__pyx_v_flags); if (unlikely(!__pyx_t_1)) {__pyx_filename =
__pyx_f[0]; __pyx_lineno = 90; __pyx_clineno = __LINE__; goto
__pyx_L1_error;}
obs and obs_a are both cdef'd np.ndarrays, and the former (obs) is
passed in as an argument.
I define flags as
cdef int flags = np.NPY_CONTIGUOUS | np.NPY_ALIGNED | np.NPY_NOTSWAPPED
and then the line that crashes is
obs_a = np.PyArray_FROM_OF(obs, flags)
Does anyone know what I'm doing wrong?
(I know I could use np.ascontiguous, but as far as I can tell this
_should_ work)
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_FROM_OF from Cython
On Thu, Oct 8, 2009 at 17:28, David Warde-Farley d...@cs.toronto.edu wrote: I'm trying to use PyArray_FROM_OF from Cython and the generated C code keeps crashing. Dag said on the Cython list that he wasn't sure what was going on, so maybe someone here will have an idea. You must call import_array() at the top level before you can use any numpy C API functions. http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_FROM_OF from Cython
On 8-Oct-09, at 6:47 PM, Robert Kern wrote: On Thu, Oct 8, 2009 at 17:28, David Warde-Farley d...@cs.toronto.edu wrote: I'm trying to use PyArray_FROM_OF from Cython and the generated C code keeps crashing. Dag said on the Cython list that he wasn't sure what was going on, so maybe someone here will have an idea. You must call import_array() at the top level before you can use any numpy C API functions. http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI Thanks. One more thing: calling Py_DECREF on arrays that I have acquired from PyArray_FROM_OF seems to cause crashes, am I correct in assuming that Cython is somehow tracking all the PyObjects in the scope (even ones acquired via the NumPy C API) and DECREF'ing it for me? David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_FROM_OF from Cython
On Thu, Oct 8, 2009 at 19:32, David Warde-Farley d...@cs.toronto.edu wrote: On 8-Oct-09, at 6:47 PM, Robert Kern wrote: On Thu, Oct 8, 2009 at 17:28, David Warde-Farley d...@cs.toronto.edu wrote: I'm trying to use PyArray_FROM_OF from Cython and the generated C code keeps crashing. Dag said on the Cython list that he wasn't sure what was going on, so maybe someone here will have an idea. You must call import_array() at the top level before you can use any numpy C API functions. http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI Thanks. One more thing: calling Py_DECREF on arrays that I have acquired from PyArray_FROM_OF seems to cause crashes, am I correct in assuming that Cython is somehow tracking all the PyObjects in the scope (even ones acquired via the NumPy C API) and DECREF'ing it for me? It usually does, yes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] byteswapping a complex scalar
On Thu, Oct 8, 2009 at 9:25 PM, Travis Oliphant oliph...@enthought.com wrote: On Oct 8, 2009, at 7:19 AM, Travis Oliphant wrote: I just checked a fix for this into SVN (tests still need to be added though...) I can't currently build SVN on my Mac for some reason (I don't know if it has to do with recent changes or not, but I don't have time to track it down right nowthe error I'm getting is something about Datetime array scalar types not being defined which seems related to the work Dave and Stefan have been discussing). I can build from SVN. The problem is I had to check-out again from SVN (and get rid of the old code-generated files --- sure would be nice if there were the equivalent of make clean Note that git clean will clean your working tree, and numscons generates less junk in the working tree. It can be a time saver and quite convenient, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
