[Numpy-discussion] List of Lists in C
Hi, Quick question, I've been doing a fair bit of extension writing in C recently, but wondered how best to implement: l = [[]] * 5 to create a list of a given length containing the initialization variable desired. A loop seems the straight forward manner, but I would have thought there was a more efficient way... Currently I just do it in Python and pass through the already initialized list as it seems perfectly efficient. Cheers, Hanni ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] List of Lists in C
Hi, Don't remember that you are using the same list in each element of the outer list. If you don't want this, use [[] for i in range(5)]. I don't think there is another way in C either (or too complicated). Matthieu 2009/1/26 Hanni Ali hanni@gmail.com: Hi, Quick question, I've been doing a fair bit of extension writing in C recently, but wondered how best to implement: l = [[]] * 5 to create a list of a given length containing the initialization variable desired. A loop seems the straight forward manner, but I would have thought there was a more efficient way... Currently I just do it in Python and pass through the already initialized list as it seems perfectly efficient. Cheers, Hanni ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] List of Lists in C
Yes fair point, but when it's a empty list and new elements are replaced with a new list instance it's fine, especially as [[]]*10 is significantly faster than [[] for i in xrange(10)] as I was previously doing. In fact I think that's partly answered my question [[]]*x must create a list of pointers pointing at the same list. Rather than [[] for i in xrange(10)] which must create a list of new separate lists instances. Hence the significant difference in speed. Hanni 2009/1/26 Matthieu Brucher matthieu.bruc...@gmail.com Hi, Don't remember that you are using the same list in each element of the outer list. If you don't want this, use [[] for i in range(5)]. I don't think there is another way in C either (or too complicated). Matthieu 2009/1/26 Hanni Ali hanni@gmail.com: Hi, Quick question, I've been doing a fair bit of extension writing in C recently, but wondered how best to implement: l = [[]] * 5 to create a list of a given length containing the initialization variable desired. A loop seems the straight forward manner, but I would have thought there was a more efficient way... Currently I just do it in Python and pass through the already initialized list as it seems perfectly efficient. Cheers, Hanni ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] List of Lists in C
2009/1/26 Hanni Ali hanni@gmail.com: Yes fair point, but when it's a empty list and new elements are replaced with a new list instance it's fine, especially as [[]]*10 is significantly faster than [[] for i in xrange(10)] as I was previously doing. In this case, why do you put a list in it in the first place ? You could put None, and it would be safer ;) In fact I think that's partly answered my question [[]]*x must create a list of pointers pointing at the same list. Rather than [[] for i in xrange(10)] which must create a list of new separate lists instances. Hence the significant difference in speed. Hanni I agree. Less memory allocations and initialization, thus more speed. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Array of matrices - Inverse and dot
Hello, I would like to operate in an easy and efficient way (without python loop) with arrays of matrices. Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would like to calculate inv_a and dot_ab, which would be arrays of N1*N2 (3*3)-matrices, such as : inv_a[i, j] = np.linalg.inv(a[i, j]) dot_ab[i, j] = np.dot(a[i, j], b[i, j]) (where a and b could be : N1 = 5 N2 = 6 a = np.random((N1, N2, 3, 3) b = np.random((N1, N2, 3, 3) ). Thank you, (Sorry to ask the list : I think it is quite a basic stuff, but searching for array of matrices on google didn't help me so much.) Jean-Baptiste Rudant ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Array of matrices - Inverse and dot
On Mon, Jan 26, 2009 at 11:59 PM, Jean-Baptiste Rudant boogalo...@yahoo.fr wrote: Hello, I would like to operate in an easy and efficient way (without python loop) with arrays of matrices. Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would like to calculate inv_a and dot_ab, which would be arrays of N1*N2 (3*3)-matrices, such as : If you only care about 3*3 matrices, I would consider using developed formula of a 3x3 matrix inverse coded in numpy. numpy.inv will be quite slow for such small matrices anyway (all the machinery to call the underlying C code being almost certainly the bottleneck). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Re : Array of matrices - Inverse and dot
On Mon, Jan 26, 2009 at 10:25 AM, Jean-Baptiste Rudant boogalo...@yahoo.fr wrote: Thank you, but my example was bad. I have to deal with matrices which can be 100*100. De : David Cournapeau courn...@gmail.com À : Discussion of Numerical Python numpy-discussion@scipy.org Envoyé le : Lundi, 26 Janvier 2009, 16h18mn 33s Objet : Re: [Numpy-discussion] Array of matrices - Inverse and dot On Mon, Jan 26, 2009 at 11:59 PM, Jean-Baptiste Rudant boogalo...@yahoo.fr wrote: Hello, I would like to operate in an easy and efficient way (without python loop) with arrays of matrices. Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would like to calculate inv_a and dot_ab, which would be arrays of N1*N2 (3*3)-matrices, such as : If you only care about 3*3 matrices, I would consider using developed formula of a 3x3 matrix inverse coded in numpy. numpy.inv will be quite slow for such small matrices anyway (all the machinery to call the underlying C code being almost certainly the bottleneck). David There is numpy.tensordot and numpy.tensorinv which looks like it might be doing what you want, but I never used it. Josef ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Academic citation ?
What is the most up-to-date way to cite Numpy and Scipy in an academic journal ? Cite our conference articles here: http://conference.scipy.org/proceedings/SciPy2008/index.html It would be nice if someone involved in the proceedings could post a bibtex on the citations page. And link the citations page to...something...easily navigated to from the front page. This brings up a related point: When someone goes to scipy.org, there is no way to navigate to conferences.scipy.org from scipy.org except by finding the link buried in the intro text. Ipython and all the whatever.scipy.org domains, except for docs.scipy.org, are completely absent; you have to know about them to find them. I don't even know where to find a complete list of these. They should all have a presence on at least the front page and maybe the navigation. --jh-- ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Academic citation ?
JH, Thx for the links, but I'm afraid I need something more basic than that. For example, I'm referring to Python as: van Rossum, G. and Drake, F. L. (eds), 2006. Python Reference Manual, Python Software Foundation,. http://docs.python.org/ref/ref.html. I could indeed use http://www.scipy.org/Citing_SciPy to cite Scipy (although the citation is incomplete), and define something similar for Numpy... Or refer to the Computing in Science and Engineering special issue. I'm just a bit surprised there's no official standard. Thx, P. On Jan 26, 2009, at 10:56 AM, j...@physics.ucf.edu wrote: What is the most up-to-date way to cite Numpy and Scipy in an academic journal ? Cite our conference articles here: http://conference.scipy.org/proceedings/SciPy2008/index.html It would be nice if someone involved in the proceedings could post a bibtex on the citations page. And link the citations page to...something...easily navigated to from the front page. This brings up a related point: When someone goes to scipy.org, there is no way to navigate to conferences.scipy.org from scipy.org except by finding the link buried in the intro text. Ipython and all the whatever.scipy.org domains, except for docs.scipy.org, are completely absent; you have to know about them to find them. I don't even know where to find a complete list of these. They should all have a presence on at least the front page and maybe the navigation. --jh-- ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Academic citation ?
@MANUAL{ascher.dubois.hinsen.hugunin.oliphant-1999-np,
author = {Ascher, David and Paul F. Dubois and Konrad Hinsen and James
Hugunin and Travis Oliphant},
year = 1999,
title= {Numerical Python},
edition = {UCRL-MA-128569},
address = {Livermore, CA},
organization = {Lawrence Livermore National Laboratory},
keywords = {numpy}
}
@ARTICLE{dubois.hinsen.hugunin-1996-cp,
author = {Dubois, Paul F. and Konrad Hinsen and James Hugunin},
year= {1996},
title = {Numerical Python},
journal = {Computers in Physics},
volume = 10,
number = 3,
month = {May/June},
keywords = {numpy}
}
@ARTICLE{dubois-1999-cse,
author = {Dubois, Paul F.},
year= 1999,
title = {Extending Python with Fortran},
journal = {Computing Science and Engineering},
volume = 1,
number = 5,
month = {Sep/Oct},
pages = {66--73},
keywords = {numpy}
}
@MANUAL{oliphant-2006-guide,
author = {Oliphant, Travis E.},
year = 2006,
title= {Guide to NumPy},
month= mar,
address = {Provo, UT},
institution = {Brigham Young University},
url = {http://www.tramy.us/},
keywords = {numpy}
}
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Array of matrices - Inverse and dot
Jean-Baptiste Rudant wrote: I would like to operate in an easy and efficient way (without python loop) with arrays of matrices. Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would like to calculate inv_a and dot_ab, which would be arrays of N1*N2 (3*3)-matrices, such as : inv_a[i, j] = np.linalg.inv(a[i, j]) dot_ab[i, j] = np.dot(a[i, j], b[i, j]) (where a and b could be : N1 = 5 N2 = 6 a = np.random((N1, N2, 3, 3) b = np.random((N1, N2, 3, 3) ). Here's a one-liner: numpy.array(map(numpy.dot, a, b)) that works for matrix multiply if a, b are (n, 3, 3). Could do the same for linalg.inv. This comes up a lot in OFDM MIMO systems so I wrote C++ code for complex matrix multiply (arbitrary size), 2x2 complex inverse and 2x2 complex matrix singular values, and then swigged it. I know a colleague at work has extended this work to arbitrary size inverse. I wish I could share the code but it is something developed for my company which has fairly strict policies about posting these things (not worth the red-tape)... I vote +1 for such features in Numpy. I haven't looked too much under the hood of numpy so I am not sure how you would do it or how hard it would be. Regards, Tom K. -- View this message in context: http://www.nabble.com/Array-of-matrices---Inverse-and-dot-tp21666949p21670624.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Academic citation ?
On 1/26/2009 12:48 PM Pierre GM apparently wrote: Shouldn't we refer to the new doc.scipy.org instead of Travis' site ? Not in my opinion: Travis wrote a book, which is what is being cited. The docs link is in fact to the same tramy site. I must add that IMO, it would be a courtesy for the docs to actually link to http://www.tramy.us/ rather than directly to the PDF. I for one urge my students to make the suggested donation when they download the book, and I am disappointed that the docs page renders this suggestion invisible. Alan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Usage of numpy.where and pylab.find with strings
Hi - I am trying to find a string in a list with strings and have come across the following state of affairs: In [228]: subjects Out[228]: ['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC', 'KFM', 'APM', 'GMC'] In [229]: subjects[0] Out[229]: 'KAA' In [230]: subjects[0] == 'KAA' Out[230]: True In [231]: np.where(subjects == 'KAA') Out[231]: () In [232]: pylab.find(subjects == 'KAA') Out[232]: array([], dtype=int32) It doesn't seem to matter if I make the list into an array: In [233]: np.array(subjects) Out[233]: array(['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC', 'KFM', 'APM', 'GMC'], dtype='|S3') In [234]: pylab.find(subjects == 'KAA') Out[234]: array([], dtype=int32) In [235]: np.where(subjects == 'KAA') Out[235]: () What am I doing wrong? What does it mean that the dtype is IS3? Thanks a bunch -- Ariel ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Usage of numpy.where and pylab.find with strings
On Mon, Jan 26, 2009 at 13:48, Ariel Rokem aro...@berkeley.edu wrote: Hi - I am trying to find a string in a list with strings and have come across the following state of affairs: In [228]: subjects Out[228]: ['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC', 'KFM', 'APM', 'GMC'] In [229]: subjects[0] Out[229]: 'KAA' In [230]: subjects[0] == 'KAA' Out[230]: True In [231]: np.where(subjects == 'KAA') Out[231]: () In [232]: pylab.find(subjects == 'KAA') Out[232]: array([], dtype=int32) Well, this will never work. Python lists don't broadcast like numpy arrays. It doesn't seem to matter if I make the list into an array: In [233]: np.array(subjects) Out[233]: array(['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC', 'KFM', 'APM', 'GMC'], dtype='|S3') In [234]: pylab.find(subjects == 'KAA') Out[234]: array([], dtype=int32) In [235]: np.where(subjects == 'KAA') Out[235]: () What am I doing wrong? subjects is still a list. You did not assign the array to that name. What does it mean that the dtype is IS3? All of your elements are length-3 strings. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Academic citation ?
On 1/26/2009 2:37 PM Pauli Virtanen apparently wrote: Fixed, no need to be dissappointed any more. Thanks! Alan PS I hope disappointed did not sound so strong as to be discourteous. Email is tricky, and I tend to write mine with dangerous speed. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bug with mafromtxt
On Jan 24, 2009, at 6:23 PM, Ryan May wrote: Ok, thanks. I've dug a little further, and it seems like the problem is that a column of all missing values ends up as a column of all None's. When you create a (masked) array from a list of None's, you end up with an object array. On one hand I'd love for things to behave differently in this case, but on the other I understand why things work this way. Ryan, Mind giving r6434 a try? As usual, don't hesitate to report any problem. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bug with mafromtxt
Pierre GM wrote: On Jan 24, 2009, at 6:23 PM, Ryan May wrote: Ok, thanks. I've dug a little further, and it seems like the problem is that a column of all missing values ends up as a column of all None's. When you create a (masked) array from a list of None's, you end up with an object array. On one hand I'd love for things to behave differently in this case, but on the other I understand why things work this way. Ryan, Mind giving r6434 a try? As usual, don't hesitate to report any problem. Works great! Thanks for the quick fix. I had racked my brain on how to go about fixing this cleanly, but this is far simpler than what I would have done. It makes sense, since all I really needed for the masked column was something *other* than object. Thanks a lot, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Usage of numpy.where and pylab.find with strings
Doh! That's embarrassing! Thanks! On Jan 26, 2009, at 12:00 PM, Keith Goodman wrote: On Mon, Jan 26, 2009 at 11:48 AM, Ariel Rokem aro...@berkeley.edu wrote: Hi - I am trying to find a string in a list with strings and have come across the following state of affairs: In [228]: subjects Out[228]: ['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC', 'KFM', 'APM', 'GMC'] In [229]: subjects[0] Out[229]: 'KAA' In [230]: subjects[0] == 'KAA' Out[230]: True In [231]: np.where(subjects == 'KAA') Out[231]: () In [232]: pylab.find(subjects == 'KAA') Out[232]: array([], dtype=int32) It doesn't seem to matter if I make the list into an array: In [233]: np.array(subjects) Out[233]: array(['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC', 'KFM', 'APM', 'GMC'], dtype='|S3') In [234]: pylab.find(subjects == 'KAA') Out[234]: array([], dtype=int32) In [235]: np.where(subjects == 'KAA') Out[235]: () What am I doing wrong? What does it mean that the dtype is IS3? I think you made a typo. Try changing line 233 to subjects = np.array(subjects) type(subjects) type 'list' np.where(subjects == 'KAA') () np.where(np.asarray(subjects) == 'KAA') (array([0]),) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy ndarray questions
Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. I think I understand the reason for this is that python does a copy internally (it does not call the __copy__ or __deepcopy__ methods?). Is there a way that I get a different object type? Or even better is there a way to prevent operations like a=a+1 or make them automatically in-place operations? I realise that I could change the __add__ ... methods but that would also prevent b=a+1 operations. My second comment is with respect to the documentation for numpy.ctypeslib.ndpointer The documentation says: An ndpointer instance is used to describe an ndarray in restypes and argtypes specifications. however if I specify a restype to a function e.g. clib.malloc.restype = ctypeslib.ndpointer(flags='contiguous,aligned',shape=shape,dtype=dtype) Calling clib.malloc(1024) results in a TypeError: TypeError: default __new__ takes no parameters Am I correct in assuming that the documentation is incorrect and ndpointer can only be used for argtypes? Or am I missing something? Cheers Jochen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. This might help some: http://www.scipy.org/Subclasses Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
On Mon, 2009-01-26 at 19:25 -0600, Ryan May wrote: Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. This might help some: http://www.scipy.org/Subclasses Ryan Thanks, I had read about __array_finalize__, but not about __array_wrap__. I'm not quite sure if I understand how I can use this to get around my problem, can you explain? Cheers Jochen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
Jochen wrote: On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote: Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. I can't comment about subclassing ndarrays, but I can give you a hint about aligned allocator problem: you could maintain two list of cached plans, automatically detect whether your arrays are aligned or not, and use the appropriate list of plans; one list is for aligned arrays, one for unaligned. Before removing support for fftw, I played with some C++ code to do exactly that. You can tell fftw to create plans for unaligned arrays by using FFTW_UNALIGNED flag: http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx cheers, David Hi David, I have actually kept more closely to the fftw way of doing things, i.e. I create a plan python object from two arrays, it also stores the two arrays to prevent someone to delete the original arrays and then executing the plan. I am not sure I follow you when you say the fftw way: you can avoid having to store the arrays altogether while still using fftw plans, there is nothing unfftw about using plans that way. I think trying to guarantee that your arrays data buffers won't change is more complicated. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote: Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. I can't comment about subclassing ndarrays, but I can give you a hint about aligned allocator problem: you could maintain two list of cached plans, automatically detect whether your arrays are aligned or not, and use the appropriate list of plans; one list is for aligned arrays, one for unaligned. Before removing support for fftw, I played with some C++ code to do exactly that. You can tell fftw to create plans for unaligned arrays by using FFTW_UNALIGNED flag: http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx cheers, David Hi David, I have actually kept more closely to the fftw way of doing things, i.e. I create a plan python object from two arrays, it also stores the two arrays to prevent someone to delete the original arrays and then executing the plan. I am not sure I follow you when you say the fftw way: you can avoid having to store the arrays altogether while still using fftw plans, there is nothing unfftw about using plans that way. I think trying to guarantee that your arrays data buffers won't change is more complicated. David Sorry maybe I wasn't quite clear, what I mean by the fftw way is creating a plan and executing the plan, instead of doing x=fft(y). As you say the problem comes when you execute a plan on arrays which don't exist anymore, which causes python to segfault (I'm talking about using fftw_execute() not fftw_execute_dft). So yes essentially my problem is trying to ensure that the buffer does not change. BTW memmap arrays have the same problem if I create a memmap array and later do something like a=a+1 all later changes will not be written to the file. BTW I just answered to you in the other thread on scipy-users as well, I'll try to just keep the rest of my posts here so people don't have to look at two lists if they want to follow things. Cheers Jochen __ _ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
Jochen wrote: On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote: Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. I can't comment about subclassing ndarrays, but I can give you a hint about aligned allocator problem: you could maintain two list of cached plans, automatically detect whether your arrays are aligned or not, and use the appropriate list of plans; one list is for aligned arrays, one for unaligned. Before removing support for fftw, I played with some C++ code to do exactly that. You can tell fftw to create plans for unaligned arrays by using FFTW_UNALIGNED flag: http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx cheers, David Hi David, I have actually kept more closely to the fftw way of doing things, i.e. I create a plan python object from two arrays, it also stores the two arrays to prevent someone to delete the original arrays and then executing the plan. I am not sure I follow you when you say the fftw way: you can avoid having to store the arrays altogether while still using fftw plans, there is nothing unfftw about using plans that way. I think trying to guarantee that your arrays data buffers won't change is more complicated. David Sorry maybe I wasn't quite clear, what I mean by the fftw way is creating a plan and executing the plan, instead of doing x=fft(y). I guess I was not very clear, because my suggestion has nothing to do with the above. As you say the problem comes when you execute a plan on arrays which don't exist anymore, which causes python to segfault (I'm talking about using fftw_execute() not fftw_execute_dft). So yes essentially my problem is trying to ensure that the buffer does not change. I agree that if you just use fftw_execute, you will have segfaults: I am just not sure that your problem is to ensure that the buffer does not change :) You can instead handle relatively easily the case where the buffers are not aligned, while still using fftw API. The fftw_execute is just not an appropriate API for python code, IMHO. Another solution would be to work on numpy itself, so that it use aligned buffers, but that's obviously much longer term - that's just one of those things on my TODO list that I never took the time to do, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
On Tue, 2009-01-27 at 13:54 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote: Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. I can't comment about subclassing ndarrays, but I can give you a hint about aligned allocator problem: you could maintain two list of cached plans, automatically detect whether your arrays are aligned or not, and use the appropriate list of plans; one list is for aligned arrays, one for unaligned. Before removing support for fftw, I played with some C++ code to do exactly that. You can tell fftw to create plans for unaligned arrays by using FFTW_UNALIGNED flag: http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx cheers, David Hi David, I have actually kept more closely to the fftw way of doing things, i.e. I create a plan python object from two arrays, it also stores the two arrays to prevent someone to delete the original arrays and then executing the plan. I am not sure I follow you when you say the fftw way: you can avoid having to store the arrays altogether while still using fftw plans, there is nothing unfftw about using plans that way. I think trying to guarantee that your arrays data buffers won't change is more complicated. David Sorry maybe I wasn't quite clear, what I mean by the fftw way is creating a plan and executing the plan, instead of doing x=fft(y). I guess I was not very clear, because my suggestion has nothing to do with the above. As you say the problem comes when you execute a plan on arrays which don't exist anymore, which causes python to segfault (I'm talking about using fftw_execute() not fftw_execute_dft). So yes essentially my problem is trying to ensure that the buffer does not change. I agree that if you just use fftw_execute, you will have segfaults: I am just not sure that your problem is to ensure that the buffer does not change :) You can instead handle relatively easily the case where the buffers are not aligned, while still using fftw API. The fftw_execute is just not an appropriate API for python code, IMHO. Ah ok, I think I understand you now :). I agree that using fftw_execute is not the most pythonic way of doing things. However every other way I could think of, you would need to do a number of checks and possibly create new plans, as well as keeping old plans around when doing a fft. (I believe that's what you did in the old fftw bindings in scipy?). So I decided to create fftw bindings first for maximum performance (I know, just doing the whole calculation in c is probably more appropriate if you want maximum performance ;), it also fits my needs. However I've been thinking about ways to make the whole thing more intuitive. At least now I can compare how much overhead I'm adding if I start to do checks in order to make the handling easier :) Another solution would be to work on numpy itself, so that it use aligned buffers, but that's obviously much longer term - that's just one of those things on my TODO list that I never took the time to do, I agree that would be nice, I'd think a couple of operations would probably benefit from that. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
Jochen wrote: On Tue, 2009-01-27 at 13:54 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote: Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. I can't comment about subclassing ndarrays, but I can give you a hint about aligned allocator problem: you could maintain two list of cached plans, automatically detect whether your arrays are aligned or not, and use the appropriate list of plans; one list is for aligned arrays, one for unaligned. Before removing support for fftw, I played with some C++ code to do exactly that. You can tell fftw to create plans for unaligned arrays by using FFTW_UNALIGNED flag: http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx cheers, David Hi David, I have actually kept more closely to the fftw way of doing things, i.e. I create a plan python object from two arrays, it also stores the two arrays to prevent someone to delete the original arrays and then executing the plan. I am not sure I follow you when you say the fftw way: you can avoid having to store the arrays altogether while still using fftw plans, there is nothing unfftw about using plans that way. I think trying to guarantee that your arrays data buffers won't change is more complicated. David Sorry maybe I wasn't quite clear, what I mean by the fftw way is creating a plan and executing the plan, instead of doing x=fft(y). I guess I was not very clear, because my suggestion has nothing to do with the above. As you say the problem comes when you execute a plan on arrays which don't exist anymore, which causes python to segfault (I'm talking about using fftw_execute() not fftw_execute_dft). So yes essentially my problem is trying to ensure that the buffer does not change. I agree that if you just use fftw_execute, you will have segfaults: I am just not sure that your problem is to ensure that the buffer does not change :) You can instead handle relatively easily the case where the buffers are not aligned, while still using fftw API. The fftw_execute is just not an appropriate API for python code, IMHO. Ah ok, I think I understand you now :). I agree that using fftw_execute is not the most pythonic way of doing things. However every other way I could think of, you would need to do a number of checks and possibly create new plans, as well as keeping old plans around when doing a fft. Yes, but I don't see the problem with keeping old plans/creating new ones. It was a pain in C++, because I had to handle many backends, not just fftw, but in your case, doing it in python is not very difficult I think. (I believe that's what you did in the old fftw bindings in scipy?). that's what was done in scipy.fftpack originally, but not by me. So I decided to create fftw bindings first for maximum performance (I know, just doing the whole calculation in c is probably more appropriate if you want maximum performance ;), it also fits my needs. Caching plans will not affect performances; on the contrary, you will be able to use more aggressively optimized plans if you add the ability to save/load plans to the disk (using wisdow mechanism). Checking for aligned arrays is one line in python, and should not affect much performances, specially if you do fft on big arrays, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy ndarray questions
On Tue, 2009-01-27 at 14:46 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 13:54 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote: Jochen wrote: On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote: Jochen wrote: Hi all, I just wrote ctypes bindings to fftw3 (see http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html for the post to scipy). Now I have a couple of numpy related questions: In order to be able to use simd instructions I create an ndarray subclass, which uses fftw_malloc to allocate the memory and fftw_free to free the memory when the array is deleted. This works fine for inplace operations however if someone does something like this: a = fftw3.AlignedArray(1024,complex) a = a+1 a.ctypes.data points to a different memory location (this is actually an even bigger problem when executing fftw plans), however type(a) still gives me class 'fftw3.planning.AlignedArray'. I can't comment about subclassing ndarrays, but I can give you a hint about aligned allocator problem: you could maintain two list of cached plans, automatically detect whether your arrays are aligned or not, and use the appropriate list of plans; one list is for aligned arrays, one for unaligned. Before removing support for fftw, I played with some C++ code to do exactly that. You can tell fftw to create plans for unaligned arrays by using FFTW_UNALIGNED flag: http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx cheers, David Hi David, I have actually kept more closely to the fftw way of doing things, i.e. I create a plan python object from two arrays, it also stores the two arrays to prevent someone to delete the original arrays and then executing the plan. I am not sure I follow you when you say the fftw way: you can avoid having to store the arrays altogether while still using fftw plans, there is nothing unfftw about using plans that way. I think trying to guarantee that your arrays data buffers won't change is more complicated. David Sorry maybe I wasn't quite clear, what I mean by the fftw way is creating a plan and executing the plan, instead of doing x=fft(y). I guess I was not very clear, because my suggestion has nothing to do with the above. As you say the problem comes when you execute a plan on arrays which don't exist anymore, which causes python to segfault (I'm talking about using fftw_execute() not fftw_execute_dft). So yes essentially my problem is trying to ensure that the buffer does not change. I agree that if you just use fftw_execute, you will have segfaults: I am just not sure that your problem is to ensure that the buffer does not change :) You can instead handle relatively easily the case where the buffers are not aligned, while still using fftw API. The fftw_execute is just not an appropriate API for python code, IMHO. Ah ok, I think I understand you now :). I agree that using fftw_execute is not the most pythonic way of doing things. However every other way I could think of, you would need to do a number of checks and possibly create new plans, as well as keeping old plans around when doing a fft. Yes, but I don't see the problem with keeping old plans/creating new ones. It was a pain in C++, because I had to handle many backends, not just fftw, but in your case, doing it in python is not very difficult I think. (I believe that's what you did in the old fftw bindings in scipy?). that's what was done in scipy.fftpack originally, but not by me. So I decided to create fftw bindings first for maximum performance (I know, just doing the whole calculation in c is probably more appropriate if you want maximum performance ;), it also fits my needs. Caching plans will not affect performances; on the contrary, you will be able to use more aggressively optimized plans if you add the ability to save/load plans to the disk (using wisdow mechanism). Checking for aligned arrays is one line in python, and should not affect much performances, specially if you do fft on big arrays, I agree checking for aligned arrays does not affect performance much, I guess the lookup mechanism for the plans is what would take up the most time IMO. I'll definitely look into this though :) Cheers Jochen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
