[Numpy-discussion] arccosh for complex numbers, goofy choice of branch
OK, Which branch do we want to use. As it currently is in numpy and scipy.special arccosh(1.5) = 0.96242365011920694 arccosh(1.5+0j) = -0.96242365011920705 + 0.0j This is consistent with gsl, but inconsistent with Mathematica, NAG, Maple, and probably all sensible implementations which use the generally accepted principal value. I've left this inconsistency raising an error in the ufunc tests until we make a decision. It might be nice to know what FORTRAN and MatLab do with this. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] arccosh for complex numbers, goofy choice of branch
Matlab is consistent, I'm afraid: acosh(1.5) ans = 0.9624 acosh(1.5 + 0j) ans = 0.9624 L. On Mon, Mar 17, 2008 at 9:40 AM, Charles R Harris [EMAIL PROTECTED] wrote: OK, Which branch do we want to use. As it currently is in numpy and scipy.special arccosh(1.5) = 0.96242365011920694 arccosh(1.5+0j) = -0.96242365011920705 + 0.0j This is consistent with gsl, but inconsistent with Mathematica, NAG, Maple, and probably all sensible implementations which use the generally accepted principal value. I've left this inconsistency raising an error in the ufunc tests until we make a decision. It might be nice to know what FORTRAN and MatLab do with this. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Lorenzo Bolla [EMAIL PROTECTED] http://lorenzobolla.emurse.com/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] arccosh for complex numbers, goofy choice of branch
On Mon, Mar 17, 2008 at 4:02 AM, lorenzo bolla [EMAIL PROTECTED] wrote: Matlab is consistent, I'm afraid: acosh(1.5) ans = 0.9624 acosh(1.5 + 0j) ans = 0.9624 OK, that does it. I'm going to change it's behavior. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to build a series of arrays as I go?
Hi All, I'm using xlrd to read an excel workbook containing several columns of data as follows: for r in range(1,sheet.nrows): date = \ datetime(*xlrd.xldate_as_tuple(sheet.cell_value(r,0),book.datemode)) if date_cut_off and date date_cut_off: continue for c in range(len(names)): name = names[c] cell = sheet.cell(r,c) if cell.ctype==xlrd.XL_CELL_EMPTY: value = -1 elif cell.ctype==xlrd.XL_CELL_DATE: value = \ datetime(*xlrd.xldate_as_tuple(cell.value,book.datemode)) else: value = cell.value data[name].append(value) Two questions: How can I build arrays as I go instead of lists? (ie: the last line of the above snippet) Once I've built arrays, how can I mask the empty cells? (the above shows my hack-so-far of turning empty cells into -1 so I can use masked_where, but it would be greato build a masked array as I went, for efficiencies sake) cheers for any help! Chris PS: Slightly pissed off at actually paying for the book only to be told it'll be 2 days before I can even read the online version, especially given the woefully inadequate state of the currently available free documentation :-( -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to build a series of arrays as I go?
Hi Chris, 1-)You could use the concatenate function to grow an array as you go. 2-) assumnig you still have your list b=numpy.array(data[name]) bmasked=numpy.ma.masked_equal(b,-1) Chris Withers wrote: Hi All, I'm using xlrd to read an excel workbook containing several columns of data as follows: for r in range(1,sheet.nrows): date = \ datetime(*xlrd.xldate_as_tuple(sheet.cell_value(r,0),book.datemode)) if date_cut_off and date date_cut_off: continue for c in range(len(names)): name = names[c] cell = sheet.cell(r,c) if cell.ctype==xlrd.XL_CELL_EMPTY: value = -1 elif cell.ctype==xlrd.XL_CELL_DATE: value = \ datetime(*xlrd.xldate_as_tuple(cell.value,book.datemode)) else: value = cell.value data[name].append(value) Two questions: How can I build arrays as I go instead of lists? (ie: the last line of the above snippet) Once I've built arrays, how can I mask the empty cells? (the above shows my hack-so-far of turning empty cells into -1 so I can use masked_where, but it would be greato build a masked array as I went, for efficiencies sake) cheers for any help! Chris PS: Slightly pissed off at actually paying for the book only to be told it'll be 2 days before I can even read the online version, especially given the woefully inadequate state of the currently available free documentation :-( ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to build a series of arrays as I go?
On Mon, 17 Mar 2008, Chris Withers apparently wrote: woefully inadequate state of the currently available free documentation 1. http://www.scipy.org/Numpy_Example_List_With_Doc 2. write some Cheers, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Read array from file
lorenzo bolla wrote: what about numpy.loadtxt? or, probably faster, the little-known (it seems) numpy.fromfile() text mode: # Read and write the first information lines for i in range(0,5): Fdif.write( Fpst.readline() ) # Read and write coordinates coords =numpy.fromfile(Fpst, dtype=numpy.float, sep=' ', count=nnod*3) coords.reshape((nnod,3)) By the way, perhaps instead of overloading numpy.fromfile(), perhaps we should just have a separate numpy.fromtextfile() function. Maybe more people would notice it. One more note: Even without fromfile() or loadtxt(), you can simplify your code. Python's duck typing and numpy's array-orientation remove a number of steps: for i in range(0,nnod): # Read line x = Fref.readline() # Read lines x = x.split() # Split line to strings x = map ( float,x ) # Convert string elements to floats no need to make an array -- numpy can work with all python sequences. #x = array ( x ) # Make an array # no need to loop == numpy assignment works with sequences: #for j in range (0,3): coords[i,:] = x Or, if you like putting code on one line (and list comprehensions): for i in range(nnod): coords[i,:] = [float(x) for x in Fref.readline.split()] -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and OpenMP
Plus a certain amount of numpy code depends on order of evaluation: a[:-1] = 2*a[1:] I'm confused here. My understanding of how it now works is that the above translates to: 1) create a new array (call it temp1) from a[1:], which shares a's data block. 2) create a temp2 array by multiplying 2 times each of the elements in temp1, and writing them into a new array, with a new data block 3) copy that temporary array into a[:-1] Why couldn't step (2) be parallelized? Why isn't it already with, BLAS? Doesn't BLAS must have such simple routines? Also, maybe numexpr could benefit from this? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to build a series of arrays as I go?
Charles Doutriaux wrote: 1-)You could use the concatenate function to grow an array as you go. Thanks. Would it be more efficient to build the whole set of arrays as lists first or build them as arrays and use concatenate? 2-) assumnig you still have your list b=numpy.array(data[name]) bmasked=numpy.ma.masked_equal(b,-1) Excellent, although I ended up using numpy.nan just ot be paranoid, in case -1 actually showed up in my data... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to build a series of arrays as I go?
Alan G Isaac wrote: On Mon, 17 Mar 2008, Chris Withers apparently wrote: woefully inadequate state of the currently available free documentation 1. http://www.scipy.org/Numpy_Example_List_With_Doc Yeah, read that, wood, trees, can't tell the... 2. write some Small problem with that... I need to understand things before I can do that, and I need docs to be able to understand... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and OpenMP
On Mon, Mar 17, 2008 at 12:06 PM, Christopher Barker [EMAIL PROTECTED] wrote: Plus a certain amount of numpy code depends on order of evaluation: a[:-1] = 2*a[1:] I'm confused here. My understanding of how it now works is that the above translates to: 1) create a new array (call it temp1) from a[1:], which shares a's data block. 2) create a temp2 array by multiplying 2 times each of the elements in temp1, and writing them into a new array, with a new data block 3) copy that temporary array into a[:-1] Why couldn't step (2) be parallelized? Why isn't it already with, BLAS? Doesn't BLAS must have such simple routines? Yes, but they are rarely optimized. We only (optionally) use the BLAS to accelerate dot(). Using the BLAS in more fundamental parts of numpy would be problematic from a build standpoint (or conversely a code complexity standpoint if it remains optional). Also, maybe numexpr could benefit from this? Possibly. You can answer this definitively by writing the code to try it out. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.ma bug: need sanity check in masked_where
File a ticket. On Mon, Mar 17, 2008 at 12:26 PM, Eric Firing [EMAIL PROTECTED] wrote: Pierre, I just tripped over what boils down to the sequence given below. It would be useful if the error in line 53 were trapped right away; as it is, it results in a masked array that looks reasonable but fails in a non-obvious way. Eric In [52]:x = [1,2] In [53]:y = ma.masked_where(False, x) In [54]:y Out[54]: masked_array(data = [1 2], mask = False, fill_value=99) In [55]:y[1] --- IndexErrorTraceback (most recent call last) /home/efiring/ipython console in module() /usr/local/lib/python2.5/site-packages/numpy/ma/core.pyc in __getitem__(self, indx) 1307 if not getattr(dout,'ndim', False): 1308 # Just a scalar - 1309 if m is not nomask and m[indx]: 1310 return masked 1311 else: IndexError: 0-d arrays can't be indexed ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and OpenMP
A Monday 17 March 2008, Christopher Barker escrigué: Plus a certain amount of numpy code depends on order of evaluation: a[:-1] = 2*a[1:] I'm confused here. My understanding of how it now works is that the above translates to: 1) create a new array (call it temp1) from a[1:], which shares a's data block. 2) create a temp2 array by multiplying 2 times each of the elements in temp1, and writing them into a new array, with a new data block 3) copy that temporary array into a[:-1] Why couldn't step (2) be parallelized? Why isn't it already with, BLAS? Doesn't BLAS must have such simple routines? Probably yes, but the problem is that this kind of operations, namely, vector-to-vector (usually found in the BLAS1 subset of BLAS), are normally memory-bounded, so you can take little avantage from using BLAS, most specially in modern processors, where the gap between the CPU throughput and the memory bandwith is quite high (and increasing). In modern machines, the use of BLAS is more interesting in vector-matrix (BLAS2) computations, but definitely is in matrix-matrix (BLAS3) ones (which is where the oportunities for cache reuse is higher) where the speedups can really be very good. Also, maybe numexpr could benefit from this? Maybe, but unfortunately it wouldn't be able to achieve high speedups. Right now, numexpr is focused in accelerating mainly vector-vector operations (or matrix-matrix, but element-wise, much like NumPy, so that the cache cannot be reused), with some smart optimizations for strided and unaligned arrays (in this scenario, it can be 2x or 3x faster than NumPy, even for very simple operations like 'a+b'). In a similar way, OpenMP (or whatever parallel paradigm) will only generally be useful when you have to deal with lots of data, and your algorithm can have the oportunity to structure it so that small portions of them can be reused many times. Cheers, -- 0,0 Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data - ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Scipy to MyHDL!
To Whom It May Concern, Please allow me to introduce myself. My name is David Allen Blubaugh. I am currently in the developmental stages of a Field-Programmable-Gate-Array (FPGA) device for a high-performance computing application. I am currently evaluating the MyHDL environment for translating python source code to verilog. I am also wondering as to what would be necessary to interface both Scipy and Numpy to the MyHDL environment? I believe that there will definitely be the need for modifications done within Numpy framework in order to quickly prototype an algorithm, like the FFT, and have it translated to verilog. Do you have any additional suggestions? Thanks, David Blubaugh This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.ma bug: need sanity check in masked_where
Charles R Harris wrote: File a ticket. #703 Eric On Mon, Mar 17, 2008 at 12:26 PM, Eric Firing [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Pierre, I just tripped over what boils down to the sequence given below. It would be useful if the error in line 53 were trapped right away; as it is, it results in a masked array that looks reasonable but fails in a non-obvious way. Eric In [52]:x = [1,2] In [53]:y = ma.masked_where(False, x) In [54]:y Out[54]: masked_array(data = [1 2], mask = False, fill_value=99) In [55]:y[1] --- IndexErrorTraceback (most recent call last) /home/efiring/ipython console in module() /usr/local/lib/python2.5/site-packages/numpy/ma/core.pyc in __getitem__(self, indx) 1307 if not getattr(dout,'ndim', False): 1308 # Just a scalar - 1309 if m is not nomask and m[indx]: 1310 return masked 1311 else: IndexError: 0-d arrays can't be indexed ___ Numpy-discussion mailing list Numpy-discussion@scipy.org mailto:Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and OpenMP
On Mon, Mar 17, 2008 at 1:59 PM, Gnata Xavier [EMAIL PROTECTED] wrote: Francesc Altet wrote: A Monday 17 March 2008, Christopher Barker escrigué: Plus a certain amount of numpy code depends on order of evaluation: a[:-1] = 2*a[1:] I'm confused here. My understanding of how it now works is that the above translates to: 1) create a new array (call it temp1) from a[1:], which shares a's data block. 2) create a temp2 array by multiplying 2 times each of the elements in temp1, and writing them into a new array, with a new data block 3) copy that temporary array into a[:-1] Why couldn't step (2) be parallelized? Why isn't it already with, BLAS? Doesn't BLAS must have such simple routines? Probably yes, but the problem is that this kind of operations, namely, vector-to-vector (usually found in the BLAS1 subset of BLAS), are normally memory-bounded, so you can take little avantage from using BLAS, most specially in modern processors, where the gap between the CPU throughput and the memory bandwith is quite high (and increasing). In modern machines, the use of BLAS is more interesting in vector-matrix (BLAS2) computations, but definitely is in matrix-matrix (BLAS3) ones (which is where the oportunities for cache reuse is higher) where the speedups can really be very good. Also, maybe numexpr could benefit from this? Maybe, but unfortunately it wouldn't be able to achieve high speedups. Right now, numexpr is focused in accelerating mainly vector-vector operations (or matrix-matrix, but element-wise, much like NumPy, so that the cache cannot be reused), with some smart optimizations for strided and unaligned arrays (in this scenario, it can be 2x or 3x faster than NumPy, even for very simple operations like 'a+b'). In a similar way, OpenMP (or whatever parallel paradigm) will only generally be useful when you have to deal with lots of data, and your algorithm can have the oportunity to structure it so that small portions of them can be reused many times. Cheers, Well, linear alagera is another topic. What I can see from IDL (for innstance) is that it provides the user with a TOTAL function which take avantage of several CPU when the number of elements is large. It also provides a very simple way to set a max number of threads. I really really would like to see something like that in numpy (just to be able to tell somone switch to numpy it is free and you will get exactly the same). For now, I have a problem when they ask for // functions like TOTAL. For now, we can do that using C inline threaded code but it is *complex* and 2000x2000 images are now common. It is not a corner case any more. Image processing may be a special in that many cases it is almost embarrassingly parallel. Perhaps some special libraries for that sort of application could be put together and just bits of c code be run on different processors. Not that I know much about parallel processing, but that would be my first take. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to build a series of arrays as I go?
Alan suggested: 1. http://www.scipy.org/Numpy_Example_List_With_Doc On Mon, 17 Mar 2008, Chris Withers apparently wrote: Yeah, read that, wood, trees, can't tell the... Oh, then you might want http://www.scipy.org/Tentative_NumPy_Tutorial or the other stuff at http://www.scipy.org/Documentation All in all, I've found the resources quite good. Cheers, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy to MyHDL!
On Mon, Mar 17, 2008 at 3:17 PM, Blubaugh, David A. [EMAIL PROTECTED] wrote: To Whom It May Concern, Please allow me to introduce myself. My name is David Allen Blubaugh. I am currently in the developmental stages of a Field-Programmable-Gate-Array (FPGA) device for a high-performance computing application. I am currently evaluating the MyHDL environment for translating python source code to verilog. I am also wondering as to what would be necessary to interface both Scipy and Numpy to the MyHDL environment? I believe that there will definitely be the need for modifications done within Numpy framework in order to quickly prototype an algorithm, like the FFT, and have it translated to verilog. Do you have any additional suggestions? Can you sketch out in more detail exactly what you are envisioning? My gut feeling is that there is very little direct interfacing that can be fruitfully done. numpy and scipy provide much higher level abstractions than MyHDL provides. I don't think there is even going to be a good way to translate those abstractions to MyHDL. One programs for silicon in an HDL rather differently than one programs for a modern microprocessor in a VHLL. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] View ND Homogeneous Record Array as (N+1)D Array?
Is there a way to view an N-dimensional array with a *homogeneous*
record dtype as an array of N+1 dimensions? An example will make it
clear:
import numpy
a = numpy.array([(1.0,2.0), (3.0,4.0)], dtype=[('A',float),('B',float)])
b = a.view(...) # do something magical
print b
array([[ 1., 2.],
[ 3., 4.]])
b[0,0] = 0.0
print a
[(0.0, 2.0) (3.0, 4.0)]
Thanks,
Alex
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] View ND Homogeneous Record Array as (N+1)D Array?
On Mon, Mar 17, 2008 at 3:44 PM, Alexander Michael [EMAIL PROTECTED] wrote:
Is there a way to view an N-dimensional array with a *homogeneous*
record dtype as an array of N+1 dimensions? An example will make it
clear:
import numpy
a = numpy.array([(1.0,2.0), (3.0,4.0)], dtype=[('A',float),('B',float)])
b = a.view(...) # do something magical
print b
array([[ 1., 2.],
[ 3., 4.]])
b[0,0] = 0.0
print a
[(0.0, 2.0) (3.0, 4.0)]
Just use a.view(float) and then reshape as appropriate.
In [1]: import numpy
In [2]: a = numpy.array([(1.0,2.0), (3.0,4.0)], dtype=[('A',float),('B',float)])
In [3]: a.view(float)
Out[3]: array([ 1., 2., 3., 4.])
In [4]: b = _
In [5]: b.shape = a.shape + (-1,)
In [6]: b
Out[6]:
array([[ 1., 2.],
[ 3., 4.]])
In [7]: b[0,0] = 0.0
In [8]: a
Out[8]:
array([(0.0, 2.0), (3.0, 4.0)],
dtype=[('A', 'f8'), ('B', 'f8')])
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy-discussion Digest, Vol 18, Issue 35
Robert, What I envisioned would be a simple but quick means to develop a FFT. I have worked this issue before with others who say that the way to do it would be to convert enough of the Numpy to MyHDL, which would then allow scipy to be imported within a python program. The question is to how this would be accomplished?? It should be stated that MyHDL is pure python programming which has no fewer capabilities than standard python. If I need to elaborate more please say so!! Thanks, David Blubaugh -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Monday, March 17, 2008 4:45 PM To: numpy-discussion@scipy.org Subject: Numpy-discussion Digest, Vol 18, Issue 35 Send Numpy-discussion mailing list submissions to numpy-discussion@scipy.org To subscribe or unsubscribe via the World Wide Web, visit http://projects.scipy.org/mailman/listinfo/numpy-discussion or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than Re: Contents of Numpy-discussion digest... Today's Topics: 1. Re: Numpy and OpenMP (Gnata Xavier) 2. Scipy to MyHDL! (Blubaugh, David A.) 3. Re: numpy.ma bug: need sanity check in masked_where (Eric Firing) 4. Re: Numpy and OpenMP (Charles R Harris) 5. Re: how to build a series of arrays as I go? (Alan G Isaac) 6. Re: Scipy to MyHDL! (Robert Kern) 7. View ND Homogeneous Record Array as (N+1)D Array? (Alexander Michael) -- Message: 1 Date: Mon, 17 Mar 2008 20:59:08 +0100 From: Gnata Xavier [EMAIL PROTECTED] Subject: Re: [Numpy-discussion] Numpy and OpenMP To: Discussion of Numerical Python numpy-discussion@scipy.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Francesc Altet wrote: A Monday 17 March 2008, Christopher Barker escrigu?: Plus a certain amount of numpy code depends on order of evaluation: a[:-1] = 2*a[1:] I'm confused here. My understanding of how it now works is that the above translates to: 1) create a new array (call it temp1) from a[1:], which shares a's data block. 2) create a temp2 array by multiplying 2 times each of the elements in temp1, and writing them into a new array, with a new data block 3) copy that temporary array into a[:-1] Why couldn't step (2) be parallelized? Why isn't it already with, BLAS? Doesn't BLAS must have such simple routines? Probably yes, but the problem is that this kind of operations, namely, vector-to-vector (usually found in the BLAS1 subset of BLAS), are normally memory-bounded, so you can take little avantage from using BLAS, most specially in modern processors, where the gap between the CPU throughput and the memory bandwith is quite high (and increasing). In modern machines, the use of BLAS is more interesting in vector-matrix (BLAS2) computations, but definitely is in matrix-matrix (BLAS3) ones (which is where the oportunities for cache reuse is higher) where the speedups can really be very good. Also, maybe numexpr could benefit from this? Maybe, but unfortunately it wouldn't be able to achieve high speedups. Right now, numexpr is focused in accelerating mainly vector-vector operations (or matrix-matrix, but element-wise, much like NumPy, so that the cache cannot be reused), with some smart optimizations for strided and unaligned arrays (in this scenario, it can be 2x or 3x faster than NumPy, even for very simple operations like 'a+b'). In a similar way, OpenMP (or whatever parallel paradigm) will only generally be useful when you have to deal with lots of data, and your algorithm can have the oportunity to structure it so that small portions of them can be reused many times. Cheers, Well, linear alagera is another topic. What I can see from IDL (for innstance) is that it provides the user with a TOTAL function which take avantage of several CPU when the number of elements is large. It also provides a very simple way to set a max number of threads. I really really would like to see something like that in numpy (just to be able to tell somone switch to numpy it is free and you will get exactly the same). For now, I have a problem when they ask for // functions like TOTAL. For now, we can do that using C inline threaded code but it is *complex* and 2000x2000 images are now common. It is not a corner case any more. Xavier -- Message: 2 Date: Mon, 17 Mar 2008 16:17:56 -0400 From: Blubaugh, David A. [EMAIL PROTECTED] Subject: [Numpy-discussion] Scipy to MyHDL! To: numpy-discussion@scipy.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=us-ascii To Whom It May Concern,
Re: [Numpy-discussion] Numpy-discussion Digest, Vol 18, Issue 35
Robert, I should also further state that MyHDL is a module that converts pure python to verilog. MyHDL is just a means to handle the necessary conversion as well as the necessary simulation of python code that is being translated to verilog. Thanks, David Blubaugh -Original Message- From: Blubaugh, David A. Sent: Monday, March 17, 2008 5:11 PM To: 'numpy-discussion@scipy.org' Cc: '[EMAIL PROTECTED]' Subject: RE: Numpy-discussion Digest, Vol 18, Issue 35 Robert, What I envisioned would be a simple but quick means to develop a FFT. I have worked this issue before with others who say that the way to do it would be to convert enough of the Numpy to MyHDL, which would then allow scipy to be imported within a python program. The question is to how this would be accomplished?? It should be stated that MyHDL is pure python programming which has no fewer capabilities than standard python. If I need to elaborate more please say so!! Thanks, David Blubaugh -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Monday, March 17, 2008 4:45 PM To: numpy-discussion@scipy.org Subject: Numpy-discussion Digest, Vol 18, Issue 35 Send Numpy-discussion mailing list submissions to numpy-discussion@scipy.org To subscribe or unsubscribe via the World Wide Web, visit http://projects.scipy.org/mailman/listinfo/numpy-discussion or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than Re: Contents of Numpy-discussion digest... Today's Topics: 1. Re: Numpy and OpenMP (Gnata Xavier) 2. Scipy to MyHDL! (Blubaugh, David A.) 3. Re: numpy.ma bug: need sanity check in masked_where (Eric Firing) 4. Re: Numpy and OpenMP (Charles R Harris) 5. Re: how to build a series of arrays as I go? (Alan G Isaac) 6. Re: Scipy to MyHDL! (Robert Kern) 7. View ND Homogeneous Record Array as (N+1)D Array? (Alexander Michael) -- Message: 1 Date: Mon, 17 Mar 2008 20:59:08 +0100 From: Gnata Xavier [EMAIL PROTECTED] Subject: Re: [Numpy-discussion] Numpy and OpenMP To: Discussion of Numerical Python numpy-discussion@scipy.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Francesc Altet wrote: A Monday 17 March 2008, Christopher Barker escrigu?: Plus a certain amount of numpy code depends on order of evaluation: a[:-1] = 2*a[1:] I'm confused here. My understanding of how it now works is that the above translates to: 1) create a new array (call it temp1) from a[1:], which shares a's data block. 2) create a temp2 array by multiplying 2 times each of the elements in temp1, and writing them into a new array, with a new data block 3) copy that temporary array into a[:-1] Why couldn't step (2) be parallelized? Why isn't it already with, BLAS? Doesn't BLAS must have such simple routines? Probably yes, but the problem is that this kind of operations, namely, vector-to-vector (usually found in the BLAS1 subset of BLAS), are normally memory-bounded, so you can take little avantage from using BLAS, most specially in modern processors, where the gap between the CPU throughput and the memory bandwith is quite high (and increasing). In modern machines, the use of BLAS is more interesting in vector-matrix (BLAS2) computations, but definitely is in matrix-matrix (BLAS3) ones (which is where the oportunities for cache reuse is higher) where the speedups can really be very good. Also, maybe numexpr could benefit from this? Maybe, but unfortunately it wouldn't be able to achieve high speedups. Right now, numexpr is focused in accelerating mainly vector-vector operations (or matrix-matrix, but element-wise, much like NumPy, so that the cache cannot be reused), with some smart optimizations for strided and unaligned arrays (in this scenario, it can be 2x or 3x faster than NumPy, even for very simple operations like 'a+b'). In a similar way, OpenMP (or whatever parallel paradigm) will only generally be useful when you have to deal with lots of data, and your algorithm can have the oportunity to structure it so that small portions of them can be reused many times. Cheers, Well, linear alagera is another topic. What I can see from IDL (for innstance) is that it provides the user with a TOTAL function which take avantage of several CPU when the number of elements is large. It also provides a very simple way to set a max number of threads. I really really would like to see something like that in numpy (just to be able to tell somone switch to numpy it is free and you will get exactly the same). For now, I have a problem when they ask for //
Re: [Numpy-discussion] Numpy and OpenMP
Charles R Harris wrote: On Mon, Mar 17, 2008 at 1:59 PM, Gnata Xavier [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Francesc Altet wrote: A Monday 17 March 2008, Christopher Barker escrigué: Plus a certain amount of numpy code depends on order of evaluation: a[:-1] = 2*a[1:] I'm confused here. My understanding of how it now works is that the above translates to: 1) create a new array (call it temp1) from a[1:], which shares a's data block. 2) create a temp2 array by multiplying 2 times each of the elements in temp1, and writing them into a new array, with a new data block 3) copy that temporary array into a[:-1] Why couldn't step (2) be parallelized? Why isn't it already with, BLAS? Doesn't BLAS must have such simple routines? Probably yes, but the problem is that this kind of operations, namely, vector-to-vector (usually found in the BLAS1 subset of BLAS), are normally memory-bounded, so you can take little avantage from using BLAS, most specially in modern processors, where the gap between the CPU throughput and the memory bandwith is quite high (and increasing). In modern machines, the use of BLAS is more interesting in vector-matrix (BLAS2) computations, but definitely is in matrix-matrix (BLAS3) ones (which is where the oportunities for cache reuse is higher) where the speedups can really be very good. Also, maybe numexpr could benefit from this? Maybe, but unfortunately it wouldn't be able to achieve high speedups. Right now, numexpr is focused in accelerating mainly vector-vector operations (or matrix-matrix, but element-wise, much like NumPy, so that the cache cannot be reused), with some smart optimizations for strided and unaligned arrays (in this scenario, it can be 2x or 3x faster than NumPy, even for very simple operations like 'a+b'). In a similar way, OpenMP (or whatever parallel paradigm) will only generally be useful when you have to deal with lots of data, and your algorithm can have the oportunity to structure it so that small portions of them can be reused many times. Cheers, Well, linear alagera is another topic. What I can see from IDL (for innstance) is that it provides the user with a TOTAL function which take avantage of several CPU when the number of elements is large. It also provides a very simple way to set a max number of threads. I really really would like to see something like that in numpy (just to be able to tell somone switch to numpy it is free and you will get exactly the same). For now, I have a problem when they ask for // functions like TOTAL. For now, we can do that using C inline threaded code but it is *complex* and 2000x2000 images are now common. It is not a corner case any more. Image processing may be a special in that many cases it is almost embarrassingly parallel. yes but who likes to do that ? One trivial case : Divide a image by its mean : Compute the mean of the image Divide the image by its mean It should be 3 small lines of code no more. Using the embarrassingly parallel paradigm to compute that, I would have to store the partial results and then run another exe to read then. Ugly. ugly but very common in the proto phases. Or it can be pipes or sockets or...wait just write in C/MPI if you want to do that. Tunning this C/MPI code you will get the best performances. Ok fine. Fine but in a few months quadcores will be cheap. Using numpy, I now I never get the best performances on a multicores machine and I do not care. I just get the best performance/time_needed_to_code_that ratio, by far, and that is why IMHO numpy is great :). The problem is that on a multicore machine, this ratio is not that high because there is no way to perform s = sum(A) in a maybe-sub-obtimal but not nonocore way. Sublinear scaling (let say real life scaling) will always be better that nothing. Xavier Perhaps some special libraries for that sort of application could be put together and just bits of c code be run on different processors. Not that I know much about parallel processing, but that would be my first take. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and OpenMP
On Mon, Mar 17, 2008 at 6:03 PM, Gnata Xavier [EMAIL PROTECTED] wrote: Ok fine. Fine but in a few months quadcores will be cheap. Using numpy, I now I never get the best performances on a multicores machine and I do not care. I just get the best performance/time_needed_to_code_that ratio, by far, and that is why IMHO numpy is great :). The problem is that on a multicore machine, this ratio is not that high because there is no way to perform s = sum(A) in a maybe-sub-obtimal but not nonocore way. Sublinear scaling (let say real life scaling) will always be better that nothing. Please, by all means go for it. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
