Re: [Numpy-discussion] Slicing/selection in multiple dimensions simultaneously
Hi Robert 2009/2/27 Robert Kern robert.k...@gmail.com: a[ix_([2,3,6],range(a.shape[1]),[3,2])] If anyone knows a better way? One could probably make ix_() take slice objects, too, to generate the correct arange() in the appropriate place. I was wondering how one would implement this, since the ix_ function has no knowledge of the dimensions of a. The best I could do was to allow a[ix_[[2,3,6], :3, [3, 2]] to work (see attached patch). Cheers Stéfan From b2076197b94e3539e804685f332d2e6a769ee1ed Mon Sep 17 00:00:00 2001 From: Stefan van der Walt ste...@sun.ac.za Date: Tue, 3 Mar 2009 11:08:52 +0200 Subject: [PATCH] Allow fully specified ranges in ix_. --- numpy/lib/index_tricks.py | 77 +++-- 1 files changed, 46 insertions(+), 31 deletions(-) diff --git a/numpy/lib/index_tricks.py b/numpy/lib/index_tricks.py index fcd3909..c7dfd29 100644 --- a/numpy/lib/index_tricks.py +++ b/numpy/lib/index_tricks.py @@ -71,37 +71,51 @@ def unravel_index(x,dims): # Indices become [x/dcb % a, x/dc % b, x/d % c, x/1 % d] return tuple(x/dim_prod % dims) -def ix_(*args): - Construct an open mesh from multiple sequences. - -This function takes n 1-d sequences and returns n outputs with n -dimensions each such that the shape is 1 in all but one dimension and -the dimension with the non-unit shape value cycles through all n -dimensions. - -Using ix_() one can quickly construct index arrays that will index -the cross product. - -a[ix_([1,3,7],[2,5,8])] returns the array - -a[1,2] a[1,5] a[1,8] -a[3,2] a[3,5] a[3,8] -a[7,2] a[7,5] a[7,8] - -out = [] -nd = len(args) -baseshape = [1]*nd -for k in range(nd): -new = _nx.asarray(args[k]) -if (new.ndim != 1): -raise ValueError, Cross index must be 1 dimensional -if issubclass(new.dtype.type, _nx.bool_): -new = new.nonzero()[0] -baseshape[k] = len(new) -new = new.reshape(tuple(baseshape)) -out.append(new) -baseshape[k] = 1 -return tuple(out) +class IndexMesh: +def __call__(self, *args): +return self.__getitem__(args) + +def __getitem__(self, *args): + Construct an open mesh from multiple sequences. + +This function takes n 1-d sequences and returns n outputs with n +dimensions each such that the shape is 1 in all but one dimension and +the dimension with the non-unit shape value cycles through all n +dimensions. + +Using ix_() one can quickly construct index arrays that will index +the cross product. + +a[ix_([1,3,7],[2,5,8])] returns the array + +a[1,2] a[1,5] a[1,8] +a[3,2] a[3,5] a[3,8] +a[7,2] a[7,5] a[7,8] + +args, = args # unpack args tuple + +out = [] +nd = len(args) +baseshape = [1]*nd +for k in range(nd): +if isinstance(args[k], slice): +s = args[k] +if s.stop is None: +raise ValueError(ix_ only accepts full range + specifications) +out.append(_nx.arange(s.start or 0, s.stop, s.step)) +continue + +new = _nx.asarray(args[k]) +if (new.ndim != 1): +raise ValueError, Cross index must be 1 dimensional +if issubclass(new.dtype.type, _nx.bool_): +new = new.nonzero()[0] +baseshape[k] = len(new) +new = new.reshape(tuple(baseshape)) +out.append(new) +baseshape[k] = 1 +return tuple(out) class nd_grid(object): @@ -210,6 +224,7 @@ class nd_grid(object): def __len__(self): return 0 +ix_ = IndexMesh() mgrid = nd_grid(sparse=False) ogrid = nd_grid(sparse=True) mgrid.__doc__ = None # set in numpy.add_newdocs -- 1.5.6.3 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] RFR: fix signed/unsigned comparison warnings in numpy
Hi, A small patch to fix some last warnings (numpy almost builds warning free with -W -Wall -Wextra now). I am not sure about those (signed/unsigned casts are potentially dangerous), so I did not apply them directly. It did help me discovering a bug or two in numpy (fixed in the trunk): http://codereview.appspot.com/24043/ http://github.com/cournape/numpy/tree/unsigned_warn cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] RFR: fix signed/unsigned comparison warnings in numpy
David Cournapeau wrote: Hi, A small patch to fix some last warnings (numpy almost builds warning free with -W -Wall -Wextra now). I am not sure about those (signed/unsigned casts are potentially dangerous), so I did not apply them directly. It did help me discovering a bug or two in numpy (fixed in the trunk): http://codereview.appspot.com/24043/ I managed to screw up the link, here is the real one: http://codereview.appspot.com/24043/show thanks Stefan !, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy and python 2.6 on windows: please test
I would appreciate if people would test building numpy (trunk); in particular since some issues are moderately complex and system dependent On Vista with VS2008, numpy rev r6535, I get the following behaviour: 1. Building and installing numpy on python 2.6.1 compiled in debug mode succeeds, but 'import numpy' returns 'ImportError: No module named multiarray'. 2. Building and installing numpy using python compiled in release mode succeeds, 'import numpy' succeeds, but 'numpy.test()' crashes the interpreter. When running in a debug version of python, 'from numpy.core import multiarray' raises an ImportError, but this does not happen with the release version, where multiarray functions seem to work. Could this be related to the PyImport_Import and PyImport_ImportModule changes made in 2.6 (bottom of http://docs.python.org/whatsnew/2.6.html)? James. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] RFR: fix signed/unsigned comparison warnings in numpy
2009/3/3 David Cournapeau da...@ar.media.kyoto-u.ac.jp: David Cournapeau wrote: Hi, A small patch to fix some last warnings (numpy almost builds warning free with -W -Wall -Wextra now). I am not sure about those (signed/unsigned casts are potentially dangerous), so I did not apply them directly. It did help me discovering a bug or two in numpy (fixed in the trunk): http://codereview.appspot.com/24043/ I managed to screw up the link, here is the real one: http://codereview.appspot.com/24043/show Looks good! Thanks, David, for explaining to me why the type cast in 977 #if @unsigntyp@ 978 if(LONG_MIN (@ctype@)x (@ctype@)x LONG_MAX) 979 return PyInt_FromLong(x); 980 #else has somewhat tricky semantics. A person is never too old to learn some more C :-) Cheers Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy and python 2.6 on windows: please test
Windows debug extensions have a suffix, d. If you don't install the debug version of numpy, you can't use it with debug Python. Ah, thank you. Sorry for the newb question: how do you install the debug version (for msvc)? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] SVN and TRAC migrations starting NOW
Dear Numpy and Scipy developers, We are now starting the svn and trac migrations to new servers: - The svn repositories of both numpy and scipy are now unavailable, and should be available around 16:00 UTC (3rd March 2009). You will then be able to update/commit again. - Trac for numpy and scipy are also unavailable. We will send an email when everything will be backed up, The Scipy website administrators ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] 64-bit numpy questions?
Hi,
I've been looking at a 64-bit numpy problem we were having on Solaris:
a=numpy.zeros(0x18000,dtype='b1')
a.data
Traceback (most recent call last):
File stdin, line 1, in module
ValueError: size must be zero or positive
A working fix seemed to be this:
Index: arrayobject.c
===
--- arrayobject.c(revision 6530)
+++ arrayobject.c(working copy)
@@ -6774,7 +6774,7 @@
static PyObject *
array_data_get(PyArrayObject *self)
{
-intp nbytes;
+Py_ssize_t nbytes;
if (!(PyArray_ISONESEGMENT(self))) {
PyErr_SetString(PyExc_AttributeError, cannot get single-\
segment buffer for discontiguous array);
@@ -6782,10 +6782,10 @@
}
nbytes = PyArray_NBYTES(self);
if PyArray_ISWRITEABLE(self) {
-return PyBuffer_FromReadWriteObject((PyObject *)self, 0, (int)
nbytes);
+return PyBuffer_FromReadWriteObject((PyObject *)self, 0,
(Py_ssize_t) nbytes);
}
else {
-return PyBuffer_FromObject((PyObject *)self, 0, (int) nbytes);
+return PyBuffer_FromObject((PyObject *)self, 0, (Py_ssize_t)
nbytes);
}
}
This fix could be simpler but still illustrates the typical problem:
use of (or cast to) int rather than something pointer sized.
I can see that a lot of effort has gone into making numpy 64-bit
enabled, but I also see a number of uses of int which look like
problems on LP64 platforms. Is anyone using numpy in 64-bit
environments on a day-to-day basis? Are you using very large arrays,
i.e. over 2G in size?
Cheers,
Todd
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit numpy questions?
On Tue, Mar 03, 2009 at 11:20:19AM -0500, Todd Miller wrote: Is anyone using numpy in 64-bitenvironments on a day-to-day basis? I am. Are you using very large arrays, i.e. over 2G in size? I believe so, but I may be wrong. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit numpy questions?
On Tue, Mar 3, 2009 at 4:20 PM, Todd Miller jmil...@stsci.edu wrote: Is anyone using numpy in 64-bit environments on a day-to-day basis? Yes - linux x86_64 Are you using very large arrays, i.e. over 2G in size? I have been using arrays this size and larger (mainly sparse matrices) without any problem (except for the machine running out of memory :) Cheers Robin Cheers, Todd ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit numpy questions?
Is anyone using numpy in 64-bit environments on a day-to-day basis? Windows 2003 64 Are you using very large arrays, i.e. over 2G in size? Yes without any problems, using Python 2.6. Hanni ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit numpy questions?
Todd Miller wrote:
Hi,
I've been looking at a 64-bit numpy problem we were having on Solaris:
a=numpy.zeros(0x18000,dtype='b1')
a.data
Traceback (most recent call last):
File stdin, line 1, in module
ValueError: size must be zero or positive
A working fix seemed to be this:
Index: arrayobject.c
===
--- arrayobject.c(revision 6530)
+++ arrayobject.c(working copy)
@@ -6774,7 +6774,7 @@
static PyObject *
array_data_get(PyArrayObject *self)
{
-intp nbytes;
+Py_ssize_t nbytes;
if (!(PyArray_ISONESEGMENT(self))) {
PyErr_SetString(PyExc_AttributeError, cannot get single-\
segment buffer for discontiguous array);
@@ -6782,10 +6782,10 @@
}
nbytes = PyArray_NBYTES(self);
if PyArray_ISWRITEABLE(self) {
-return PyBuffer_FromReadWriteObject((PyObject *)self, 0, (int)
nbytes);
+return PyBuffer_FromReadWriteObject((PyObject *)self, 0,
(Py_ssize_t) nbytes);
}
else {
-return PyBuffer_FromObject((PyObject *)self, 0, (int) nbytes);
+return PyBuffer_FromObject((PyObject *)self, 0, (Py_ssize_t)
nbytes);
}
}
This fix could be simpler but still illustrates the typical problem:
use of (or cast to) int rather than something pointer sized.
This looks like a problem with the port to Python2.5 not getting all the
Python C-API changes.There is no need to change the
intp nbytes
line, but the un-necessary casting to (int) in the calls to the
PyBuffer_ should absolutely be changed at least for Python 2.5 and above.
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Proposed schedule for numpy 1.3.0
Hi, A few weeks ago, we had a discussion about the 1.3.0 release schedule, but we did not end up stating a schedule. I stand up to be the release manager for 1.3.0, and suggest the following: Beta: 15th March (only doc + severe regressions accepted after, branch trunk into 1.3.x, trunk opened for 1.4.0) RC: 23th March (nothing but build issues/blockers) Release date: 1st April If you are in the middle of something important with a lot of changes, and you don't think you can make it for 15th March, please notify ASAP, in particular for C code (the exact dates can be changed in that case). What constitutes severe regression and blocker issues is decided by the release manager. I also started to update the release notes: http://projects.scipy.org/scipy/numpy/browser/trunk/doc/release/1.3.0-notes.rst Please fill it for missing informations (in particular for documentation, and ma, which I have not followed in detail). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help on subclassing numpy.ma: __array_wrap__
Kevin,
Sorry for the delayed answer.
(a) Is MA intended to be subclassed?
Yes, that's actually the reason why the class was rewritten, to
simplify subclassing. As Josef suggested, you can check the
scikits.timeseries package that makes an extensive use of MaskedArray
as baseclass.
(b) If so, perhaps I'm missing something to make this work. Any
pointers will be appreciated.
As you've run a debugger on your sources, you must have noticed the
calls to MaskedArray._update_from. In your case, the simplest is to
define DTMA._update_from as such:
_
def _update_from(self, obj):
ma.MaskedArray._update_from(self, obj)
self._attr = getattr(obj, '_attr', {'EmptyDict':[]})
_
Now, because MaskedArray.__array_wrap__() itself calls _update_from,
you don't actually need a specific DTMA.__array_wrap__ (unless you
have some specific operations to perform, but it doesn't seem to be
the case).
Now for a word of explanation:
__array_wrap__ is intended to transform the output of a numpy function
to an object of your class. When we use the numpy.ma functions, we
don't need that, we just need to retrieve some of the attributes of
the initial MA. That's why _update_from was introduced.
Of course, I'm to blame not to have make that aspect explicit in the
doc. I gonna try to correct that.
In any case, let me know how it goes.
P.
On Mar 1, 2009, at 10:37 AM, Kevin Dunn wrote:
Hi everyone,
I'm subclassing Numpy's MaskedArray to create a data class that
handles missing data, but adds some extra info I need to carrry
around. However I've been having problems keeping this extra info
attached to the subclass instances after performing operations on
them.
The bare-bones script that I've copied here shows the basic issue:
http://pastebin.com/f69b979b8
There are 2 classes: one where I am able to subclass numpy (with
help from the great description at http://www.scipy.org/Subclasses),
and the other where I subclass numpy.ma, using the same ideas again.
When stepping through the code in a debugger, lines 76 to 96, I can
see that the numpy subclass, called DT, calls DT.__array_wrap__()
after it completes unary and binary operations. But the numpy.ma
subclass, called DTMA, does not seem to call DTMA.__array_wrap__(),
especially line 111.
Just to test this idea, I overrode the __mul__ function in my DTMA
subclass to call DTMA.__array_wrap__() and it returns my extra
attributes, in the same way that Numpy did.
My questions are:
(b) If so, perhaps I'm missing something to make this work. Any
pointers will be appreciated.
So far it seems the only way for me to sub-class numpy.ma is to
override all numpy.ma functions of interest for my class and add a
DTMA.__array_wrap() call to the end of them. Hopefully there is an
easier way.
Related to this question, was there are particular outcome from this
archived discussion (I only joined the list recently):
http://article.gmane.org/gmane.comp.python.numeric.general/24315
because that dictionary object would be exactly what I'm after here.
Thanks,
Kevin
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] SVN and Trac servers are back up
Hi everyone, We have moved the scipy and numpy Trac and SVN servers to a new machine. We have also moved the scikits SVN repository, but not its Trac (scipy.org/scipy/scikits). The SVN repositories for wavelets, mpi4py, and other projects that are hosted on scipy have not been moved yet, and will be temporarily unavailable until we get them moved over. Please poke around (gently!) and let us know if you experience any broken links, incorrect redirects, and the like. A few things to note: - The URLs for the trac pages have been simplified to: http://projects.scipy.org/numpy http://projects.scipy.org/scipy You should be seemlessly redirected to these sites if you try to access any of the old URLs (which were of the form /scipy/scipy/ or / scipy/numpy/). - The mailman archives and listinfo pages should now redirect to mail.scipy.org/mailman/ and mail.scipy.org/pipermail/. Again, this should be seemless, so if you experience any difficulties please let us know. Thanks, Peter, Stefan, and David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVN and Trac servers are back up
Hi Peter, Peter Wang wrote: Hi everyone, We have moved the scipy and numpy Trac and SVN servers to a new machine. We have also moved the scikits SVN repository, but not its Trac (scipy.org/scipy/scikits). The SVN repositories for wavelets, mpi4py, and other projects that are hosted on scipy have not been moved yet, and will be temporarily unavailable until we get them moved over. Please poke around (gently!) and let us know if you experience any broken links, incorrect redirects, and the like. A few things to note: - The URLs for the trac pages have been simplified to: http://projects.scipy.org/numpy http://projects.scipy.org/scipy You should be seemlessly redirected to these sites if you try to access any of the old URLs (which were of the form /scipy/scipy/ or / scipy/numpy/). It looks like modifying tickets still cause some trouble: after clicking submit changes, the server seems to hang, and the request never ends, Thanks again for all your work on the migration, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy and python 2.6 on windows: please test
Windows debug extensions have a suffix, d. If you don't install the debug version of numpy, you can't use it with debug Python. *red face* forgot about --debug... Yes, this has actually nothing to do with python 2.6. I noticed the crash, thought naively it would be easy to fix, but it is actually quite nasty. I've reverted the change which introduced the crash for the time being (r6541). In case this helps, there are only 2 test lines in a single test in r6535 which crash the interpreter on python-2.6.1-win32: numpy/ma/tests/test_mrecords.py, in test_get(), - assert_equal(mbase_first.tolist(), (1,1.1,'one')) - assert_equal(mbase_last.tolist(), (None,None,None)) mbase[int].tolist() crashes Python in _Py_Dealloc(PyObject *op). mbase.tolist() does not. I am very unfamiliar with how debugging works on the python + windows + VS combination. If you have some insight/recommendations, I would be glad to fix this (e.g. how is this supposed to work ?) I'm not sure if there's a better way, but I've found it easiest to run python via a debug run from within VS, installing and testing numpy from there. The 2.6.1 sources build fine with VS2008. James. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed schedule for numpy 1.3.0
David, I also started to update the release notes: http://projects.scipy.org/scipy/numpy/browser/trunk/doc/release/1.3.0-notes.rst I get a 404. Anyhow, on the ma side: * structured arrays should now be fully supported by MaskedArray (r6463, r6324, r6305, r6300, r6294...) * Minor bug fixes (r6356, r6352, r6335, r6299, r6298) * Improved support for __iter__ (r6326) * made baseclass, sharedmask and hardmask accesible to the user (but read-only) + doc update ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed schedule for numpy 1.3.0
Pierre GM wrote: David, I also started to update the release notes: http://projects.scipy.org/scipy/numpy/browser/trunk/doc/release/1.3.0-notes.rst I get a 404. Anyhow, on the ma side: * structured arrays should now be fully supported by MaskedArray (r6463, r6324, r6305, r6300, r6294...) * Minor bug fixes (r6356, r6352, r6335, r6299, r6298) * Improved support for __iter__ (r6326) * made baseclass, sharedmask and hardmask accesible to the user (but read-only) + doc update I also received a 404. However it is available in svn (numpy/doc/release). Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVN and Trac servers are back up
2009/3/3 Peter Wang pw...@enthought.com: Can you try again? I looked again and it looks like there are definitely some files that were not writeable by the Apache server. I think it is the notification system that is failing to send out e-mails. I'll look into it. Cheers Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed schedule for numpy 1.3.0
On Wed, Mar 4, 2009 at 4:52 AM, Pierre GM pgmdevl...@gmail.com wrote: David, I also started to update the release notes: http://projects.scipy.org/scipy/numpy/browser/trunk/doc/release/1.3.0-notes.rst I get a 404. Yes, sorry about that, the server migration has caused some changes here. I will wait for the DNS update to get up to my connection. Anyway, the file is in svn, though, so feel free to update it accordingly. thanks for the info on masked arrays, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVN and Trac servers are back up
2009/3/3 Stéfan van der Walt ste...@sun.ac.za: 2009/3/3 Peter Wang pw...@enthought.com: Can you try again? I looked again and it looks like there are definitely some files that were not writeable by the Apache server. I think it is the notification system that is failing to send out e-mails. I'll look into it. Should be fixed now. Thanks Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [Fwd: Issue 113 in sphinx: nodes.py KeyError entries in numpy doc build]
I have been trying to build numpy docs from the svn doc subdirectory, and I have been notifying the sphinx people when things don't work. They have made several fixes, but here is one that will require a change on the numpy side. I know zip about sphinx extensions, so I don't expect to be able to suggest a fix myself. (I am assuming that the doc subdirectory in svn *should* be buildable--am I correct?) Eric ---BeginMessage--- Issue 113: nodes.py KeyError entries in numpy doc build http://bitbucket.org/birkenfeld/sphinx/issue/113/nodespy-keyerror-entries-in-numpy-doc Georg Brandl / birkenfeld on Tue, 3 Mar 2009 21:23:46 +0100: Comment: I'm afraid this is something the numpy people have to fix -- the entries toctree attribute is new in 0.6. Changes: status: new - wontfix -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. ---End Message--- ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Fwd: Issue 113 in sphinx: nodes.py KeyError entries in numpy doc build]
Tue, 03 Mar 2009 11:15:31 -1000, Eric Firing wrote: I have been trying to build numpy docs from the svn doc subdirectory, and I have been notifying the sphinx people when things don't work. They have made several fixes, but here is one that will require a change on the numpy side. I know zip about sphinx extensions, so I don't expect to be able to suggest a fix myself. (I am assuming that the doc subdirectory in svn *should* be buildable--am I correct?) Yes. But not necessarily with Sphinx 0.6.dev. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] intersect1d and setmember1d
Robert Kern robert.kern at gmail.com writes: Do you mind if we just add you to the THANKS.txt file, and consider you as a NumPy Developer per the LICENSE.txt as having released that code under the numpy license? If we're dotting our i's and crossing our t's legally, that's a bit more straightforward (oddly enough). No, I don't mind having it released under the numpy licence. Neil ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Faster way to generate a rotation matrix?
Hi, I am doing optimization on a vector of rotation angles tx,ty and tz using scipy.optimize.fmin. Unfortunately the function that I am optimizing needs the rotation matrix corresponding to this vector so it is getting constructed once for each iteration with new values. From profiling I can Hi, I am doing optimization on a vector of rotation angles tx,ty and tz using scipy.optimize.fmin. Unfortunately the function that I am optimizing needs the rotation matrix corresponding to this vector so it is getting constructed once for each iteration with new values. From profiling I can see that the function I am using to construct this rotation matrix is a bottleneck. I am currently using: def rotation(theta): tx,ty,tz = theta Rx = np.array([[1,0,0], [0, cos(tx), -sin(tx)], [0, sin(tx), cos(tx)]]) Ry = np.array([[cos(ty), 0, -sin(ty)], [0, 1, 0], [sin(ty), 0, cos(ty)]]) Rz = np.array([[cos(tz), -sin(tz), 0], [sin(tz), cos(tz), 0], [0,0,1]]) return np.dot(Rx, np.dot(Ry, Rz)) Is there a faster way to do this? Perhaps I can do this faster with a small cython module, but this might be overkill? Thanks for any help, Jonathan.see that the function I am using to construct this rotation matrix is a bottleneck. I am currently using: def rotation(theta): tx,ty,tz = theta Rx = np.array([[1,0,0], [0, cos(tx), -sin(tx)], [0, sin(tx), cos(tx)]]) Ry = np.array([[cos(ty), 0, -sin(ty)], [0, 1, 0], [sin(ty), 0, cos(ty)]]) Rz = np.array([[cos(tz), -sin(tz), 0], [sin(tz), cos(tz), 0], [0,0,1]]) return np.dot(Rx, np.dot(Ry, Rz)) Is there a faster way to do this? Perhaps I can do this faster with a small cython module, but this might be overkill? Thanks for any help, Jonathan. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
Sorry.. obviously having some copy and paste trouble here. The message should be as follows: Hi, I am doing optimization on a vector of rotation angles tx,ty and tz using scipy.optimize.fmin. Unfortunately the function that I am optimizing needs the rotation matrix corresponding to this vector so it is getting constructed once for each iteration with new values. From profiling I can see that the function I am using to construct this rotation matrix is a bottleneck. I am currently using: def rotation(theta): tx,ty,tz = theta Rx = np.array([[1,0,0], [0, cos(tx), -sin(tx)], [0, sin(tx), cos(tx)]]) Ry = np.array([[cos(ty), 0, -sin(ty)], [0, 1, 0], [sin(ty), 0, cos(ty)]]) Rz = np.array([[cos(tz), -sin(tz), 0], [sin(tz), cos(tz), 0], [0,0,1]]) return np.dot(Rx, np.dot(Ry, Rz)) Is there a faster way to do this? Perhaps I can do this faster with a small cython module, but this might be overkill? Thanks for any help, Jonathan. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
On Tue, Mar 3, 2009 at 17:53, Jonathan Taylor jonathan.tay...@utoronto.ca wrote: Sorry.. obviously having some copy and paste trouble here. The message should be as follows: Hi, I am doing optimization on a vector of rotation angles tx,ty and tz using scipy.optimize.fmin. Unfortunately the function that I am optimizing needs the rotation matrix corresponding to this vector so it is getting constructed once for each iteration with new values. From profiling I can see that the function I am using to construct this rotation matrix is a bottleneck. I am currently using: def rotation(theta): tx,ty,tz = theta Rx = np.array([[1,0,0], [0, cos(tx), -sin(tx)], [0, sin(tx), cos(tx)]]) Ry = np.array([[cos(ty), 0, -sin(ty)], [0, 1, 0], [sin(ty), 0, cos(ty)]]) Rz = np.array([[cos(tz), -sin(tz), 0], [sin(tz), cos(tz), 0], [0,0,1]]) return np.dot(Rx, np.dot(Ry, Rz)) Is there a faster way to do this? Perhaps I can do this faster with a small cython module, but this might be overkill? You could look up to the full form of the rotation matrix in terms of the angles, or use sympy to do the same. The latter might be more convenient given that the reference you find might be using a different convention for the angles. James Diebel's Representing Attitude: Euler Angles, Unit Quaternions, and Rotation Vectors is a nice, comprehensive reference for such formulae. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=5F5145BE25D61F87478B25AD1493C8F4?doi=10.1.1.110.5134rep=rep1type=pdfei=QcetSefqF4GEsQPnx4jSBAsig2=HjJILSBPFgJTfuifbvKrxwusg=AFQjCNFbABIxusr-NEbgrinhtR6buvjaYA -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
In addition to what Robert said, you also only need to calculate six transcendentals: cx = cos(tx) sx = sin(tx) cy = cos(ty) sy = sin(ty) cz = cos(tz) sz = sin(tz) you, are making sixteen transcendental calls in your loop each time. I can also recommend Chapter 2 of Introduction to Robotics: Mechanics and Controls by John J. Craig for more on more efficient transformations. On Tue, Mar 3, 2009 at 7:19 PM, Robert Kern robert.k...@gmail.com wrote: On Tue, Mar 3, 2009 at 17:53, Jonathan Taylor jonathan.tay...@utoronto.ca wrote: Sorry.. obviously having some copy and paste trouble here. The message should be as follows: Hi, I am doing optimization on a vector of rotation angles tx,ty and tz using scipy.optimize.fmin. Unfortunately the function that I am optimizing needs the rotation matrix corresponding to this vector so it is getting constructed once for each iteration with new values. From profiling I can see that the function I am using to construct this rotation matrix is a bottleneck. I am currently using: def rotation(theta): tx,ty,tz = theta Rx = np.array([[1,0,0], [0, cos(tx), -sin(tx)], [0, sin(tx), cos(tx)]]) Ry = np.array([[cos(ty), 0, -sin(ty)], [0, 1, 0], [sin(ty), 0, cos(ty)]]) Rz = np.array([[cos(tz), -sin(tz), 0], [sin(tz), cos(tz), 0], [0,0,1]]) return np.dot(Rx, np.dot(Ry, Rz)) Is there a faster way to do this? Perhaps I can do this faster with a small cython module, but this might be overkill? You could look up to the full form of the rotation matrix in terms of the angles, or use sympy to do the same. The latter might be more convenient given that the reference you find might be using a different convention for the angles. James Diebel's Representing Attitude: Euler Angles, Unit Quaternions, and Rotation Vectors is a nice, comprehensive reference for such formulae. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=5F5145BE25D61F87478B25AD1493C8F4?doi=10.1.1.110.5134rep=rep1type=pdfei=QcetSefqF4GEsQPnx4jSBAsig2=HjJILSBPFgJTfuifbvKrxwusg=AFQjCNFbABIxusr-NEbgrinhtR6buvjaYA -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Slicing/selection in multiple dimensions simultaneously
On Tue, Mar 3, 2009 at 03:11, Stéfan van der Walt ste...@sun.ac.za wrote: Hi Robert 2009/2/27 Robert Kern robert.k...@gmail.com: a[ix_([2,3,6],range(a.shape[1]),[3,2])] If anyone knows a better way? One could probably make ix_() take slice objects, too, to generate the correct arange() in the appropriate place. I was wondering how one would implement this, since the ix_ function has no knowledge of the dimensions of a. No, you're right. It doesn't work. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to multiply the rows of a matrix by a different number?
I guess there has to be an easy way for this. I have: M.shape=(1,3) N.shape=(1,) I want to do this: for i in range(1): M[i]*=N[i] without the explicit loop -Jose ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to multiply the rows of a matrix by a different number?
On Tue, Mar 3, 2009 at 8:53 PM, Jose Borreguero borregu...@gmail.com wrote: I guess there has to be an easy way for this. I have: M.shape=(1,3) N.shape=(1,) I want to do this: for i in range(1): M[i]*=N[i] without the explicit loop M = np.ones((10,3)) N = np.arange(10) N.shape (10,) (N[:,np.newaxis]).shape (10, 1) M*N[:,np.newaxis] array([[ 0., 0., 0.], [ 1., 1., 1.], [ 2., 2., 2.], [ 3., 3., 3.], [ 4., 4., 4.], [ 5., 5., 5.], [ 6., 6., 6.], [ 7., 7., 7.], [ 8., 8., 8.], [ 9., 9., 9.]]) M *= N[:,np.newaxis] M Josef ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to multiply the rows of a matrix by a different number?
M *= N[:, newaxis] On Tue, 3 Mar 2009, Jose Borreguero wrote: I guess there has to be an easy way for this. I have: M.shape=(1,3) N.shape=(1,) I want to do this: for i in range(1): M[i]*=N[i] without the explicit loop -Jose ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
On Tue, Mar 3, 2009 at 22:41, Jonathan Taylor jonathan.tay...@utoronto.ca wrote: Thanks, All these things make sense and I should have known to calculate the sins and cosines up front. I managed a few more tricks and knocked off 40% of the computation time: def rotation(theta, R = np.zeros((3,3))): cx,cy,cz = np.cos(theta) sx,sy,sz = np.sin(theta) R.flat = (cx*cz - sx*cy*sz, cx*sz + sx*cy*cz, sx*sy, -sx*cz - cx*cy*sz, -sx*sz + cx*cy*cz, cx*sy, sy*sz, -sy*cz, cy) return R Pretty evil looking ;) but still wouldn't mind somehow getting it faster. Am I right in thinking that I wouldn't get much of a speedup by rewriting this in C as most of the time is spent in necessary python functions? You would be able to get rid of the Python function call overhead of rotation(), the ufunc machinery, and unnecessary array creation/deletion. It could be worth your time experimenting with a quick Cython implementation. That might help you (and the rest of us!) answer this question instinctively the next time around. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
On Tue, Mar 3, 2009 at 11:41 PM, Jonathan Taylor jonathan.tay...@utoronto.ca wrote: Thanks, All these things make sense and I should have known to calculate the sins and cosines up front. I managed a few more tricks and knocked off 40% of the computation time: def rotation(theta, R = np.zeros((3,3))): cx,cy,cz = np.cos(theta) sx,sy,sz = np.sin(theta) R.flat = (cx*cz - sx*cy*sz, cx*sz + sx*cy*cz, sx*sy, -sx*cz - cx*cy*sz, -sx*sz + cx*cy*cz, cx*sy, sy*sz, -sy*cz, cy) return R Pretty evil looking ;) but still wouldn't mind somehow getting it faster. One of the usual recommendation on the python list is also to load functions into the local scope to avoid the lookup in the module. e.g. npcos = np.cos or I think the usual: `from numpy import cos, sin, zeros` should be better for speed also you still have a few duplicate multiplications, e.g. cx*cz, cx*sz, ..? but this looks already like micro optimization. Josef ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
since you only need to calculate the sine or cosine of a single value (not an array of values) I would recommend using the sine and cosine function of the python standard math library as it is a full order of magnitude faster. (at least on my core 2 windows vista box) i.e. import math as m m.sin m.cos etc again, this is microoptimization a difference of 10^-5s per call vs 10^-6s per call numpy vs math Chris On Wed, Mar 4, 2009 at 12:11 AM, josef.p...@gmail.com wrote: On Tue, Mar 3, 2009 at 11:41 PM, Jonathan Taylor jonathan.tay...@utoronto.ca wrote: Thanks, All these things make sense and I should have known to calculate the sins and cosines up front. I managed a few more tricks and knocked off 40% of the computation time: def rotation(theta, R = np.zeros((3,3))): cx,cy,cz = np.cos(theta) sx,sy,sz = np.sin(theta) R.flat = (cx*cz - sx*cy*sz, cx*sz + sx*cy*cz, sx*sy, -sx*cz - cx*cy*sz, -sx*sz + cx*cy*cz, cx*sy, sy*sz, -sy*cz, cy) return R Pretty evil looking ;) but still wouldn't mind somehow getting it faster. One of the usual recommendation on the python list is also to load functions into the local scope to avoid the lookup in the module. e.g. npcos = np.cos or I think the usual: `from numpy import cos, sin, zeros` should be better for speed also you still have a few duplicate multiplications, e.g. cx*cz, cx*sz, ..? but this looks already like micro optimization. Josef ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
Hello, def rotation(theta, R = np.zeros((3,3))): cx,cy,cz = np.cos(theta) sx,sy,sz = np.sin(theta) R.flat = (cx*cz - sx*cy*sz, cx*sz + sx*cy*cz, sx*sy, -sx*cz - cx*cy*sz, -sx*sz + cx*cy*cz, cx*sy, sy*sz, -sy*cz, cy) return R Pretty evil looking ;) but still wouldn't mind somehow getting it faster I would definitely encourage you to check out cython. I have to write lots of numerically intensive stuff in my python code, and I tend to cythonize it a lot. In cython, the above would be (something like): from numpy cimport ndarray cdef extern from math.h: double cos(double) double sin(double) def rotation(ndarry[double] theta, ndarray[double, ndim=2] R = np.zeros((3,3))): cdef double cx = cos(theta[0]), cy = cos(theta[1]), cz = cos(theta[2]) cdef double sx = sin(theta[0]), sy = sin(theta[1]), sz = sin(theta[2]) R[0,0] = cx*cz - sx*cy*sz R[0,1] = cx*sz + sx*cy*cz R[0,2] = sx*sy ... R[2,2] = cy return R And that will be probably be orders of magnitude faster than what you currently have, as everything but the function call and the return statement would become C code. Compilers these days are very good at optimizing that kind of thing too. --Hoyt + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoy...@gmail.com ++ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
On 3-Mar-09, at 11:41 PM, Jonathan Taylor wrote: def rotation(theta, R = np.zeros((3,3))): Hey Jon, Just a note, in case you haven't heard this schpiel before: be careful when you use mutables as default arguments. It can lead to unexpected behaviour down the line. The reason is that the np.zeros() is only called once when the function is read by the interpreter, and that reference is retained between calls.Example: def f(a=[0,0,0]): ... print sum(a) ... a[0] += 1 ... f() 0 f() 1 f() 2 In your example this is fine, since you're overwriting every single value in the function body, but if you weren't, it would lead to problems that would be difficult to debug. (On a related note never try and reverse an array in-place with slice syntax. i.e. a[:] = a[::-1] is bad mojo and cost me about a day. :P) Cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
On Wed, Mar 4, 2009 at 00:56, David Warde-Farley d...@cs.toronto.edu wrote: On 3-Mar-09, at 11:41 PM, Jonathan Taylor wrote: def rotation(theta, R = np.zeros((3,3))): Hey Jon, Just a note, in case you haven't heard this schpiel before: be careful when you use mutables as default arguments. It can lead to unexpected behaviour down the line. The reason is that the np.zeros() is only called once when the function is read by the interpreter, and that reference is retained between calls. I'm pretty sure that's exactly why he did it, and that's what he's calling evil. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
On 4-Mar-09, at 1:58 AM, Robert Kern wrote: I'm pretty sure that's exactly why he did it, and that's what he's calling evil. As ever, such nuance is lost on me. I didn't bother to check whether or not it was in the original function. Robert to the rescue. :) It's a neat trick, actually, though I'll probably shoot myself in the foot if I try to use it. I've taken to the somewhat unpythonic convention of passing in out arrays everywhere to avoid repeated allocations, i.e. def foo(x, y, out=None): if out == None: out = np.empty(...) It's a neat trick, the default argument one, but I'd probably shoot myself in the foot with it. Also, I guess it only works when the dimensions are fixed (like in this case). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Faster way to generate a rotation matrix?
On 4-Mar-09, at 1:50 AM, Hoyt Koepke wrote: I would definitely encourage you to check out cython. I have to write lots of numerically intensive stuff in my python code, and I tend to cythonize it a lot. Seconded. I recently took some distance computation code and Cythonized it, I got an absolutely absurd speedup after about five minutes of effort to remind myself how numpy+cython play together (see: http://wiki.cython.org/tutorials/numpy ). I know some people are reluctant to use anything where their code isn't standard python (possibly with ctypes fiddling) but Cython is definitely worth it for replacing bottlenecks. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
