Re: [Numpy-discussion] Some numpy statistics
On Wed, Dec 10, 2008 at 12:55 AM, [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 01:49, Charles R Harris [EMAIL PROTECTED] wrote: Hi All, I bumped into this while searching for something else: http://www.ohloh.net/p/numpy/analyses/latest -14 lines of Javascript? Well, they have scipy mostly written in C++ and davidc as a C developer with a 29000 line commit ;) The code analysis isn't quite perfect and I think there are some bugs in computing the statistics. But it's kind of interesting anyway. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Some numpy statistics
On Wed, Dec 10, 2008 at 02:22, Charles R Harris [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 12:55 AM, [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 01:49, Charles R Harris [EMAIL PROTECTED] wrote: Hi All, I bumped into this while searching for something else: http://www.ohloh.net/p/numpy/analyses/latest -14 lines of Javascript? Well, they have scipy mostly written in C++ and davidc as a C developer with a 29000 line commit ;) The code analysis isn't quite perfect and I think there are some bugs in computing the statistics. But it's kind of interesting anyway. There are bugs, and then there are bugs. It seems like an invariants numlines = 0 should pertain even with dodgy language identification. I simply don't know what operations they would do to get negative numbers. In any case, sloccount tells me that most of scipy *is* C++. The generated sparsetools sources are quite large in addition to all of the Blitz sources. SLOCDirectory SLOC-by-Language (Sorted) 177304 sparse cpp=134410,ansic=22394,fortran=12780,python=7720 96740 weave cpp=82265,python=14244,ansic=231 39321 special fortran=19749,ansic=16888,python=2684 18074 integrate fortran=15871,python=1156,ansic=1047 14472 interpolate fortran=10564,python=2493,ansic=1210,cpp=205 12471 ndimage python=6242,ansic=6229 11431 optimizefortran=5931,python=2864,ansic=2636 11390 odr fortran=9380,ansic=1192,python=818 9951stats python=8526,fortran=1425 6801signal ansic=3934,python=2867 5878fftpack fortran=3973,python=1462,ansic=443 5756io python=4987,ansic=769 4672spatial python=2731,ansic=1941 4608cluster python=2659,ansic=1949 4227linalg python=3605,fortran=604,ansic=18 1530lib python=1182,fortran=324,ansic=24 1471stsci ansic=976,python=495 1125maxentropy python=1125 940 miscpython=940 494 constants python=494 160 top_dir python=160 3 linsolvepython=3 Totals grouped by language (dominant language first): cpp: 216880 (50.58%) fortran: 80601 (18.80%) python: 69457 (16.20%) ansic:61881 (14.43%) Total Physical Source Lines of Code (SLOC)= 428,819 Development Effort Estimate, Person-Years (Person-Months) = 116.12 (1,393.47) (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05)) Schedule Estimate, Years (Months) = 3.26 (39.15) (Basic COCOMO model, Months = 2.5 * (person-months**0.38)) Estimated Average Number of Developers (Effort/Schedule) = 35.60 Total Estimated Cost to Develop = $ 15,686,619 (average salary = $56,286/year, overhead = 2.40). SLOCCount, Copyright (C) 2001-2004 David A. Wheeler SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL. SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to redistribute it under certain conditions as specified by the GNU GPL license; see the documentation for details. Please credit this data as generated using David A. Wheeler's 'SLOCCount'. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Some numpy statistics
Charles R Harris wrote: On Wed, Dec 10, 2008 at 12:55 AM, [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 01:49, Charles R Harris [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Hi All, I bumped into this while searching for something else: http://www.ohloh.net/p/numpy/analyses/latest -14 lines of Javascript? Well, they have scipy mostly written in C++ and davidc as a C developer with a 29000 line commit ;) C++ in scipy mostly is generated code (sparsetools) + blitz. There is also the problem of code reformating: for example, ohloh seems to believe I am an advanced Fortran developer from scipy, whereas I barely know how to code an hello world; I guess this is because of my removal of arpack while the license issue was discussed and solved. IIRC, I did use the svn method to put back the code, so in theory, it should be possible to realize I did not code any of the above. Also, svn is pretty dumb about renaming (it is just an atomic copy + rm), so if you remove a file, I would not be surprised if you become the author of the whole file for svn in that case. I mean, I am far from being the main author of scipy for any meaningful measure of contribution. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Little vectorization help...
Hi All, I am tryin to vectorize 3 nested for loops but I am not having much success. Here is the code I use: import numpy import numpy.ma as masked grid = numpy.zeros((nx, ny), dtype=numpy.float32) xOut = numpy.zeros((nx, ny), dtype=numpy.float32) yOut = numpy.zeros((nx, ny), dtype=numpy.float32) z = GetCentroids() # Some vector z values prop = GetValue() # Some other vector values NaN = numpy.NaN for I in xrange(1, nx+1): for J in xrange(1, ny+1): theSum = [] for K in xrange(1, nz+1): cellPos = I-1 + nx*(J-1) + nx*ny*(K-1) centroid = z[cellPos] if low = centroid = high and actnum[cellPos] 0: theSum.append(prop[cellPos]) if theSum: grid[I-1, J-1] = sum(theSum)/len(theSum) else: grid[I-1, J-1] = NaN xOut[I-1, J-1], yOut[I-1, J-1] = x[cellPos], y[cellPos] grid = masked.masked_where(numpy.isnan(grid), grid) Some explanation: 1) z is a vector of nx*ny*nz components, where nx = 100, ny = 73, nz = 23, which represents 3D hexahedron cell centroids; 2) prop is a vector like z, with the same shape, with some floating point values in it; 3) actnum is a vector of integers (0 or 1) with the same shape as z, and indicates if a cell should be considered in the loop or not; 4) low and high are 2 floating point values with low high: if the cell centroid fall between low and high and the cell is active (as stated in actnum), then I take the value of prop in that cell and I append it to the theSum list; 5) At the end of the K loop, I just take an arithmetic mean of the values in theSum list. I think I may be able to figure out how to vectorize the part regarding the grid variable, but I have no idea on what to do for the xOut and yOut variables, and I need them because I use them in a later call to matplotlib.contourf. If you could drop some hint on how to proceed, ot would be very appreciated. Thank you for your suggestions. Andrea. Imagination Is The Only Weapon In The War Against Reality. http://xoomer.alice.it/infinity77/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Little vectorization help...
Hi All, I am tryin to vectorize 3 nested for loops but I am not having much success. Here is the code I use: import numpy import numpy.ma as masked grid = numpy.zeros((nx, ny), dtype=numpy.float32) xOut = numpy.zeros((nx, ny), dtype=numpy.float32) yOut = numpy.zeros((nx, ny), dtype=numpy.float32) z = GetCentroids() # Some vector z values prop = GetValue() # Some other vector values NaN = numpy.NaN for I in xrange(1, nx+1): for J in xrange(1, ny+1): theSum = [] for K in xrange(1, nz+1): cellPos = I-1 + nx*(J-1) + nx*ny*(K-1) centroid = z[cellPos] if low = centroid = high and actnum[cellPos] 0: theSum.append(prop[cellPos]) if theSum: grid[I-1, J-1] = sum(theSum)/len(theSum) else: grid[I-1, J-1] = NaN xOut[I-1, J-1], yOut[I-1, J-1] = x[cellPos], y[cellPos] grid = masked.masked_where(numpy.isnan(grid), grid) Some explanation: 1) z is a vector of nx*ny*nz components, where nx = 100, ny = 73, nz = 23, which represents 3D hexahedron cell centroids; 2) prop is a vector like z, with the same shape, with some floating point values in it; 3) actnum is a vector of integers (0 or 1) with the same shape as z, and indicates if a cell should be considered in the loop or not; 4) low and high are 2 floating point values with low high: if the cell centroid fall between low and high and the cell is active (as stated in actnum), then I take the value of prop in that cell and I append it to the theSum list; 5) At the end of the K loop, I just take an arithmetic mean of the values in theSum list. I think I may be able to figure out how to vectorize the part regarding the grid variable, but I have no idea on what to do for the xOut and yOut variables, and I need them because I use them in a later call to matplotlib.contourf. If you could drop some hint on how to proceed, ot would be very appreciated. Thank you for your suggestions. Andrea. Imagination Is The Only Weapon In The War Against Reality. http://xoomer.alice.it/infinity77/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Superclassing numpy.matrix: got an unexpected keyword argument 'dtype'
Hello, I'm using numpy-1.1.1 for Python 2.3. I'm trying to create a class that acts just like the numpy.matrix class with my own added methods and attributes. I want to pass my class a list of custom instrument objects and do some math based on these objects to set the matrix. To this end I've done the following: from numpy import matrix class rcMatrix(matrix): def __init__(self,instruments): Do some calculations and set the values of the matrix. self[0,0] = 100 # Just an example self[0,1] = 100 # The real init method self[1,0] = 200 # Does some math based on the input objects self[1,1] = 300 # def __new__(cls,instruments): When creating a new instance begin by creating an NxN matrix of zeroes. len_ = len(instruments) return matrix.__new__(cls,[[0.0]*len_]*len_) It works great and I can, for example, multiply two of my custom matrices seamlessly. I can also get the transpose. However, when I try to get the inverse I get an error: rcm = rcMatrix(['instrument1','instrument2']) print rcm [[ 100. 100.] [ 200. 300.]] print rcm.T [[ 100. 200.] [ 100. 300.]] print [5,10] * rcm [[ 2500. 3500.]] print rcm.I Traceback (most recent call last): File [Standard]/deleteme, line 29, in ? File C:\Python23\Lib\site-packages\numpy\core\defmatrix.py, line 492, in getI return asmatrix(func(self)) File C:\Python23\Lib\site-packages\numpy\core\defmatrix.py, line 52, in asmatrix return matrix(data, dtype=dtype, copy=False) TypeError: __init__() got an unexpected keyword argument 'dtype' I've had to overwrite the getI function in order for things to work out: def getI(self): return matrix(self.tolist()).I I = property(getI, None, doc=inverse) Is this the correct way to achieve my goals? Please let me know if anything is unclear. Thanks, Robert Conde ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] access ndarray in C++
On Wednesday 23 April 2008 15:48:23 Christopher Barker wrote: - Boost Python [1]. Especially if you want usable C++ integration. (ie. more than basic templates, etc.) What's the status of the Boost array object? maintained? updated for recent numpy? The boost.python array object is still maintained. However, it has a few problems: 1. All array operations go through python which makes it too slow for my purposes. Phil Austin posted an alternate class on this list which works well since it uses the numpy C API: http://www.eos.ubc.ca/research/clouds/software/pythonlibs/num_util/num_util_release2/Readme.html 2. Only numeric numarray are supported out of the box, but it is simple to support numpy; just add the following after calling import_array in your extension module: boost::python::numeric::array::set_module_and_type( numpy, ndarray ); 3. If you want the C++-way of dealing with numpy matrices vectors directly as objects look at either of the following: http://mail.python.org/pipermail/cplusplus-sig/2008-October/013825.html http://mathema.tician.de/software/pyublas Of course, I am biased towards the first approach. Regards, Ravi ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Superclassing numpy.matrix: got an unexpected keyword argument 'dtype'
Robert, Transforming your matrix to a list before computation isn't very efficient. If you do need some extra parameters in your __init__ to be compatible with other functions such as asmatrix, well, just add them, or use a coverall **kwargs def __init__(self, instruments, **kwargs) No guarantee it'll work all the time. Otherwise, please have a look at: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html and the other link at the top of that page. In your case, I'd try to put the initialization in the __array_finalize__. On Dec 10, 2008, at 7:15 AM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hello, I'm using numpy-1.1.1 for Python 2.3. I'm trying to create a class that acts just like the numpy.matrix class with my own added methods and attributes. I want to pass my class a list of custom instrument objects and do some math based on these objects to set the matrix. To this end I've done the following: from numpy import matrix class rcMatrix(matrix): def __init__(self,instruments): Do some calculations and set the values of the matrix. self[0,0] = 100 # Just an example self[0,1] = 100 # The real init method self[1,0] = 200 # Does some math based on the input objects self[1,1] = 300 # def __new__(cls,instruments): When creating a new instance begin by creating an NxN matrix of zeroes. len_ = len(instruments) return matrix.__new__(cls,[[0.0]*len_]*len_) It works great and I can, for example, multiply two of my custom matrices seamlessly. I can also get the transpose. However, when I try to get the inverse I get an error: rcm = rcMatrix(['instrument1','instrument2']) print rcm [[ 100. 100.] [ 200. 300.]] print rcm.T [[ 100. 200.] [ 100. 300.]] print [5,10] * rcm [[ 2500. 3500.]] print rcm.I Traceback (most recent call last): File [Standard]/deleteme, line 29, in ? File C:\Python23\Lib\site-packages\numpy\core\defmatrix.py, line 492, in getI return asmatrix(func(self)) File C:\Python23\Lib\site-packages\numpy\core\defmatrix.py, line 52, in asmatrix return matrix(data, dtype=dtype, copy=False) TypeError: __init__() got an unexpected keyword argument 'dtype' I've had to overwrite the getI function in order for things to work out: def getI(self): return matrix(self.tolist()).I I = property(getI, None, doc=inverse) Is this the correct way to achieve my goals? Please let me know if anything is unclear. Thanks, Robert Conde ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] access ndarray in C++
Oops, please ignore my previous message. I just started using a new mail client which marked some of my old messages (which I had tagged interesting) the same as new messages and I just blindly replied to them without checking the date. Sorry about the spam. Ravi ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Find index of repeated numbers in array
Hi Everyone I think I'm missing something really obvious but what I would like to do is extract the indexes from an array where a number matches - For example data = [0,1,2,960,5,6,960,7] I would like to know, for example the indices which match 960 - i.e. it would return 3 and 6 I could do this with a loop but I was wondering if there was a built in numpy function to do this? BTW if anyone is interested I'm converting some idl code to numpy and trying to mmic the IDL function where Cheers Ross ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find index of repeated numbers in array
Ross Williamson wrote: Hi Everyone I think I'm missing something really obvious but what I would like to do is extract the indexes from an array where a number matches - For example data = [0,1,2,960,5,6,960,7] I would like to know, for example the indices which match 960 - i.e. it would return 3 and 6 import numpy as np In[14]: np.where( np.array( data ) == 960 ) Out[14]: (array([3, 6]),) If you need to count all of the items, try something like np.histogram( data, np.max( data ) ) cheers, r. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find index of repeated numbers in array
On Wed, 10 Dec 2008 11:38:11 -0500 Ross Williamson [EMAIL PROTECTED] wrote: Hi Everyone I think I'm missing something really obvious but what I would like to do is extract the indexes from an array where a number matches - For example data = [0,1,2,960,5,6,960,7] I would like to know, for example the indices which match 960 - i.e. it would return 3 and 6 I could do this with a loop but I was wondering if there was a built in numpy function to do this? BTW if anyone is interested I'm converting some idl code to numpy and trying to mmic the IDL function where Cheers Ross ___ Numpy-discussion mailing list Numpy-discussion@scipy.org data array([ 0, 1, 2, 960, 5, 6, 960, 7]) where(data==960) (array([3, 6]),) Nils ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] genloadtxt : last call
Pierre GM wrote: in the same place in NumPy; and all the SciPy IO code to be in the same place in SciPy. +1 So, no problem w/ importing numpy.ma and numpy.records in numpy.lib.io ? As long as numpy.ma and numpy.records are, and will remain, part of the standard numpy distribution, this is fine. This is a key issue -- what is core numpy and what is not, but I know I'd like to see a lot of things built on ma and records, both, so I think they do belong in core. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find index of repeated numbers in array
On 12/10/2008 5:38 PM, Ross Williamson wrote: Hi Everyone I think I'm missing something really obvious but what I would like to do is extract the indexes from an array where a number matches - For example data = [0,1,2,960,5,6,960,7] I would like to know, for example the indices which match 960 - i.e. it would return 3 and 6 import numpy a = numpy.array([0,1,2,960,5,6,960,7]) a == 960 array([False, False, False, True, False, False, True, False], dtype=bool) idx, = numpy.where(a == 960) idx array([3, 6]) idx.tolist() [3, 6] Sturla Molden ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANNOUNCE: ETS 3.1.0 released!
I'm pleased to announce that the Enthought Tool Suite (ETS) 3.1.0 has been tagged, released, and uploaded to PyPi[1]! Both source distributions (.tar.gz) and binary (.egg) for Windows have been built and uploaded to PyPi. You can update an existing ETS install to v3.1.0 like so: easy_install -U ETS==3.1.0 What is ETS? -- The Enthought Tool Suite (ETS) is a collection of projects developed by members of the OSS community, including Enthought employees, which we use every day to construct custom scientific applications. It includes a wide variety of components, including: * an extensible application framework * application building blocks * 2-D and 3-D graphics libraries * scientific and math libraries * developer tools The cornerstone on which these tools rest is the Traits project, which provides explicit type declarations in Python; its features include initialization, validation, delegation, notification, and visualization of typed attributes. More information is available for all these packages from the Enthought Tool Suite development home page: http://code.enthought.com/projects/index.php -- Dave ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how do I delete unused matrix to save the memory?
I am running in ipython. Now I do not have the problem anymore. %reset commands is a good solution. Thanks Frank Date: Tue, 9 Dec 2008 21:03:00 -0600 From: [EMAIL PROTECTED] To: numpy-discussion@scipy.org Subject: Re: [Numpy-discussion] how do I delete unused matrix to save the memory? On Mon, Dec 8, 2008 at 19:15, frank wang [EMAIL PROTECTED] wrote: Hi, I have a program with some variables consume a lot of memory. The first time I run it, it is fine. The second time I run it, I will get MemoryError. If I close the ipython and reopen it again, then I can run the program once. I am looking for a command to delete the intermediate variable once it is not used to save memory like in matlab clear command. How are you running this program? Be aware that IPython may be holding on to objects and preventing them from being deallocated. For example: In [7]: !cat memtest.py class A(object): def __del__(self): print 'Deleting %r' % self a = A() In [8]: %run memtest.py In [9]: %run memtest.py In [10]: %run memtest.py In [11]: del a In [12]: Do you really want to exit ([y]/n)? $ python memtest.py Deleting __main__.A object at 0x915ab0 You can remove some of these references with %reset and maybe a gc.collect() for good measure. In [1]: %run memtest In [2]: %run memtest In [3]: %run memtest In [4]: %reset Once deleted, variables cannot be recovered. Proceed (y/[n])? y Deleting __main__.A object at 0xf3e950 Deleting __main__.A object at 0xf3e6d0 Deleting __main__.A object at 0xf3e930 -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion _ Send e-mail faster without improving your typing skills. http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Importance of order when summing values in anarray
On my two systems with Intel Core2 DUO, finfo(float128) gives me the nameerro, NameError: name 'float128' is not defined. Why? Thanks Frank Date: Tue, 9 Dec 2008 21:10:32 -0600 From: [EMAIL PROTECTED] To: numpy-discussion@scipy.org Subject: Re: [Numpy-discussion] Importance of order when summing values in anarray On Tue, Dec 9, 2008 at 21:01, Charles R Harris [EMAIL PROTECTED] wrote:On Tue, Dec 9, 2008 at 1:40 PM, Robert Kern [EMAIL PROTECTED] wrote: On Tue, Dec 9, 2008 at 09:51, Nadav Horesh [EMAIL PROTECTED] wrote: As much as I know float128 are in fact 80 bits (64 mantissa + 16 exponent) so the precision is 18-19 digits (not 34) float128 should be 128 bits wide. If it's not on your platform, please let us know as that is a bug in your build. I think he means the actual precision is the ieee extended precision, the number just happens to be stored into larger chunks of memory for alignment purposes. Ah, that's good to know. Yes, float128 on my Intel Mac behaves this way. In [12]: f = finfo(float128) In [13]: f.nmant Out[13]: 63 In [14]: f.nexp Out[14]: 15 -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion _ You live life online. So we put Windows on the web. http://clk.atdmt.com/MRT/go/127032869/direct/01/___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Importance of order when summing values in anarray
On Wed, Dec 10, 2008 at 11:00 AM, frank wang [EMAIL PROTECTED] wrote: On my two systems with Intel Core2 DUO, finfo(float128) gives me the nameerro, NameError: name 'float128' is not defined. Why? You probably run a 32 bit OS. IEEE extended precision is 80 bits. On 32 bit systems it fits in three 32 bit words and shows up as float96. On 64 bit systems it fits in two 64 bit words and shows up as float128. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find index of repeated numbers in array
Thanks all I was being dumb and forgot to initialize as array() Cheers Ross On Dec 10, 2008, at 11:47 AM, Sturla Molden wrote: On 12/10/2008 5:38 PM, Ross Williamson wrote: Hi Everyone I think I'm missing something really obvious but what I would like to do is extract the indexes from an array where a number matches - For example data = [0,1,2,960,5,6,960,7] I would like to know, for example the indices which match 960 - i.e. it would return 3 and 6 import numpy a = numpy.array([0,1,2,960,5,6,960,7]) a == 960 array([False, False, False, True, False, False, True, False], dtype=bool) idx, = numpy.where(a == 960) idx array([3, 6]) idx.tolist() [3, 6] Sturla Molden ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Importance of order when summing values in anarray
On Wed, Dec 10, 2008 at 12:07, Charles R Harris [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 11:00 AM, frank wang [EMAIL PROTECTED] wrote: On my two systems with Intel Core2 DUO, finfo(float128) gives me the nameerro, NameError: name 'float128' is not defined. Why? You probably run a 32 bit OS. IEEE extended precision is 80 bits. On 32 bit systems it fits in three 32 bit words and shows up as float96. On 64 bit systems it fits in two 64 bit words and shows up as float128. I'm running a 32-bit OS (well, a 32-bit build of Python on OS X) on an Intel Core2 Duo, and I get a float128. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] On the quality of the numpy.random.normal() distribution
Hello, I have been reading that there may be potential issues with the Box-Muller transform, which is used by the numpy.random.normal() function. Supposedly, since f*x1 and f*x2 are not independent variables, then the individual elements (corresponding to f*x1 and f*x2 ) of the distribution also won't be independent. For example, see Stochastic Simulation by Ripley, pages 54-59, where the random values end up distributed on a spiral. Note that they mention that they only looked at congruential generators. Is the random number generator used by numpy congruential? I have tried to generate plots that demonstrate this problem, but have come up short. For example: import numpy , pylab nsamples = 10**6 n = numpy.random.normal( 0.0 , 1.0 , nsamples ) pylab.scatter( n[0:-1:2] , n[1:-1:2] , 0.1 ) pylab.show() I can zoom in and out, and the scatter still looks random (white noise -- almost like tv static). Does this prove that there is no problem? And if so, why does numpy do a better job than as demonstrated by Ripley? Regards, Mike Gilbert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On the quality of the numpy.random.normal() distribution
I think the use of a correct uniform generator will allow a good normal distribution. Congruental generators are very basic generators, everyone knows they should not be used. I think Numpy uses a Mersenne Twisted generator, for which you can generate independant vectors with several hundred values. Matthieu 2008/12/10 Michael Gilbert [EMAIL PROTECTED]: Hello, I have been reading that there may be potential issues with the Box-Muller transform, which is used by the numpy.random.normal() function. Supposedly, since f*x1 and f*x2 are not independent variables, then the individual elements (corresponding to f*x1 and f*x2 ) of the distribution also won't be independent. For example, see Stochastic Simulation by Ripley, pages 54-59, where the random values end up distributed on a spiral. Note that they mention that they only looked at congruential generators. Is the random number generator used by numpy congruential? I have tried to generate plots that demonstrate this problem, but have come up short. For example: import numpy , pylab nsamples = 10**6 n = numpy.random.normal( 0.0 , 1.0 , nsamples ) pylab.scatter( n[0:-1:2] , n[1:-1:2] , 0.1 ) pylab.show() I can zoom in and out, and the scatter still looks random (white noise -- almost like tv static). Does this prove that there is no problem? And if so, why does numpy do a better job than as demonstrated by Ripley? Regards, Mike Gilbert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On the quality of the numpy.random.normal() distribution
Wed, 10 Dec 2008 14:03:39 -0500, Michael Gilbert wrote: I have been reading that there may be potential issues with the Box-Muller transform, which is used by the numpy.random.normal() function. Supposedly, since f*x1 and f*x2 are not independent variables, then the individual elements (corresponding to f*x1 and f*x2 ) of the distribution also won't be independent. For example, see Stochastic Simulation by Ripley, pages 54-59, where the random values end up distributed on a spiral. Note that they mention that they only looked at congruential generators. Is the random number generator used by numpy congruential? I'm not an expert, but the generator used by Numpy is the Mersenne twister, which should be quite good for many uses. I'd guess what you mention is a way to illustrate that the output of linear congruental generators has serial correlations. At least according to wikipedia, these are negligible in Mersenne twister's output. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Importance of order when summing values in anarray
On Wed, Dec 10, 2008 at 11:58 AM, Robert Kern [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 12:07, Charles R Harris [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 11:00 AM, frank wang [EMAIL PROTECTED] wrote: On my two systems with Intel Core2 DUO, finfo(float128) gives me the nameerro, NameError: name 'float128' is not defined. Why? You probably run a 32 bit OS. IEEE extended precision is 80 bits. On 32 bit systems it fits in three 32 bit words and shows up as float96. On 64 bit systems it fits in two 64 bit words and shows up as float128. I'm running a 32-bit OS (well, a 32-bit build of Python on OS X) on an Intel Core2 Duo, and I get a float128. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Importance of order when summing values in anarray
On Wed, Dec 10, 2008 at 11:58 AM, Robert Kern [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 12:07, Charles R Harris [EMAIL PROTECTED] wrote: On Wed, Dec 10, 2008 at 11:00 AM, frank wang [EMAIL PROTECTED] wrote: On my two systems with Intel Core2 DUO, finfo(float128) gives me the nameerro, NameError: name 'float128' is not defined. Why? You probably run a 32 bit OS. IEEE extended precision is 80 bits. On 32 bit systems it fits in three 32 bit words and shows up as float96. On 64 bit systems it fits in two 64 bit words and shows up as float128. I'm running a 32-bit OS (well, a 32-bit build of Python on OS X) on an Intel Core2 Duo, and I get a float128. Curious. It probably has something to do with the way the FPU is set up when running on a 64 bit system that is independent of how python is compiled. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On the quality of the numpy.random.normal() distribution
On Wed, Dec 10, 2008 at 12:03 PM, Michael Gilbert [EMAIL PROTECTED] wrote: Hello, I have been reading that there may be potential issues with the Box-Muller transform, which is used by the numpy.random.normal() function. Supposedly, since f*x1 and f*x2 are not independent variables, then the individual elements (corresponding to f*x1 and f*x2 ) of the distribution also won't be independent. For example, see Stochastic Simulation by Ripley, pages 54-59, where the random values end up distributed on a spiral. Note that they mention that they only looked at congruential generators. Is the random number generator used by numpy congruential? I have tried to generate plots that demonstrate this problem, but have come up short. For example: import numpy , pylab nsamples = 10**6 n = numpy.random.normal( 0.0 , 1.0 , nsamples ) pylab.scatter( n[0:-1:2] , n[1:-1:2] , 0.1 ) pylab.show() I can zoom in and out, and the scatter still looks random (white noise -- almost like tv static). Does this prove that there is no problem? And if so, why does numpy do a better job than as demonstrated by Ripley? Bruce Carneal did some tests of robustness and speed for various normal generators. I don't know what his final tests showed for Box-Muller. IIRC, it had some failures but nothing spectacular. The tests were pretty stringent and based on using the erf to turn the normal distribution into a uniform distribution and using the crush tests on the latter.. You could send him a note and ask: [EMAIL PROTECTED] Here are the timings he got: In what follows the uniform variate generators are: lcg64 mwc8222 mt19937 mt19937_64 yarn5 And the normal distribution codes are: trng - default normal distribution code in TRNG boxm - Box-Muller, mtrand lookalike, remembers/uses 2nd value zig7 - a 'Harris' ziggurat indexed by 7 bits zig8 - a 'Harris' ziggurat indexed by 8 bits zig9 - a 'Harris' ziggurat indexed by 9 bits Here are the numbers in more detail: # Timings from icc -O2 running on 2.4GhZ Core-2 lcg64 trng: 6.52459e+06 ops per second lcg64 boxm: 2.18453e+07 ops per second lcg64 zig7: 1.80616e+08 ops per second lcg64 zig8: 2.01865e+08 ops per second lcg64 zig9: 2.05156e+08 ops per second mwc8222 trng: 6.52459e+06 ops per second mwc8222 boxm: 2.08787e+07 ops per second mwc8222 zig7: 9.44663e+07 ops per second mwc8222 zig8: 1.05326e+08 ops per second mwc8222 zig9: 1.03478e+08 ops per second mt19937 trng: 6.41112e+06 ops per second mt19937 boxm: 1.64986e+07 ops per second mt19937 zig7: 4.23762e+07 ops per second mt19937 zig8: 4.52623e+07 ops per second mt19937 zig9: 4.52623e+07 ops per second mt19937_64 trng: 6.42509e+06 ops per second mt19937_64 boxm: 1.93226e+07 ops per second mt19937_64 zig7: 5.8762e+07 ops per second mt19937_64 zig8: 6.17213e+07 ops per second mt19937_64 zig9: 6.29146e+07 ops per second yarn5 trng: 5.95781e+06 ops per second yarn5 boxm: 1.19156e+07 ops per second yarn5 zig7: 1.48945e+07 ops per second yarn5 zig8: 1.54809e+07 ops per second yarn5 zig9: 1.53201e+07 ops per second # Timings from g++ -O2 running on a 2.4GhZ Core-2 lcg64 trng: 6.72163e+06 ops per second lcg64 boxm: 1.50465e+07 ops per second lcg64 zig7: 1.31072e+08 ops per second lcg64 zig8: 1.48383e+08 ops per second lcg64 zig9: 1.6036e+08 ops per second mwc8222 trng: 6.64215e+06 ops per second mwc8222 boxm: 1.44299e+07 ops per second mwc8222 zig7: 8.903e+07 ops per second mwc8222 zig8: 1.00825e+08 ops per second mwc8222 zig9: 1.03478e+08 ops per second mt19937 trng: 6.52459e+06 ops per second mt19937 boxm: 1.28223e+07 ops per second mt19937 zig7: 5.00116e+07 ops per second mt19937 zig8: 5.41123e+07 ops per second mt19937 zig9: 5.47083e+07 ops per second mt19937_64 trng: 6.58285e+06 ops per second mt19937_64 boxm: 1.42988e+07 ops per second mt19937_64 zig7: 6.72164e+07 ops per second mt19937_64 zig8: 7.39591e+07 ops per second mt19937_64 zig9: 7.46022e+07 ops per second yarn5 trng: 6.25144e+06 ops per second yarn5 boxm: 8.93672e+06 ops per second yarn5 zig7: 1.50465e+07 ops per second yarn5 zig8: 1.57496e+07 ops per second yarn5 zig9: 1.56038e+07 ops per second Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] rollaxis and reshape
Hi I'm trying to split an array into two pieces and have the two pieces in a new dimension. Here it is in code, because that's hard to explain in words. data.shape (4, 50, 3) new_data = numpy.zeros((2, 4, 25, 3)) new_data[0,...] = data[:,:25,:] new_data[1,...] = data[:,25:,:] new_data.shape (2, 4, 25, 3) That works but when I try it with reshape the elements get in the wrong place. I've tried various combinations of rollaxis before the reshape, but can't get it right. Thanks Eleanor -- View this message in context: http://www.nabble.com/rollaxis-and-reshape-tp20943690p20943690.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Failing to build numpy properly on Ubuntu Hardy
Hi all, Looks like I am following the long line of people failing to build numpy :). I must admit I am clueless with building problems. Numpy builds alright, but I get: ImportError: /usr/lib/sse2/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done On import. This used to work a while ago. I am not sure what I changed, but it sure does fail. I really don't understand where the gfortran comes in. I tried building numpy with or without gfortran. From what I gather it is the numpy is being built by a different compiler than the atlas libraries (hurray for ABI compatibility), but I don't really understand how this is possible. How can I debug this? Cheers, Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Support for sparse matrix in Distance function (and clustering)?
Damian Eads eads at soe.ucsc.edu writes: Hi, Can you be more specific? Do you need sparse matrices to represent observation vectors because they are sparse? Or do you need sparse matrices to represent distance matrices because most vectors you are clustering are similar while a few are dissimilar? Damian On Tue, Dec 9, 2008 at 1:28 PM, Bab Tei babaktei at yahoo.com wrote: Hi Does the distance function in spatial package support sparse matrix? regards Hi I need sparse matrices to represent observation vectors because they are sparse. I have a large sparse matrix. I also use kmeans (Besides hierarchical clustering) which can directly work with very large data. Teimourpour ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Excluding index in numpy like negative index in R?
Keith Goodman kwgoodman at gmail.com writes: On Tue, Dec 9, 2008 at 12:25 PM, Bab Tei babaktei at yahoo.com wrote: I can exclude a list of items by using negative index in R (R-project) ie myarray[-excludeindex]. As negative indexing in numpy (And python) behave differently ,how can I exclude a list of item in numpy? Here's a painful way to do it: x = np.array([0,1,2,3,4]) excludeindex = [1,3] idx = list(set(range(4)) - set(excludeindex)) x[idx] array([0, 2]) To make it more painful, you might want to sort idx. But if excludeindex is True/False, then just use ~excludeindex. Thank you. However it seems I have to create a full list at first and then exclude items. It is somehow painful as I have some very large sparse matrices and creating a full index eats a lot of memory. Maybe adding this functionality to numpy saves memory and makes the syntax more clear ie a syntax like x[~excludeindex] which smartly distinguish between excludeindex as a list of numerical indexes and a mask (list of true/false indexes). Regards ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Failing to build numpy properly on Ubuntu Hardy
On Wed, Dec 10, 2008 at 11:10:23PM +0100, Gael Varoquaux wrote: Numpy builds alright, but I get: ImportError: /usr/lib/sse2/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done Doh! I knew it must be a FAQ, and it was :). Better googling gave me the answer: the configuration was picking up the libraries for the libatlas3gf-sse2 package, which is built with gfortran. Numpy is built with g77, and I need to force it to link with the libraries given by the atlas3-sse2 package (providing libaries built with g77). The best way is simply to remove the gfortran altas libraries. This email from David got me on the track: http://projects.scipy.org/pipermail/numpy-discussion/2008-May/034164.html I must have at some point installed the gfortran libraries by mistake. I was taken by surprise because I didn't expect Ubuntu to have 2 versions of atlas, ABI incompatible. Sorry for the noise. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Failing to build numpy properly on Ubuntu Hardy
Gael Varoquaux wrote: I must have at some point installed the gfortran libraries by mistake. I was taken by surprise because I didn't expect Ubuntu to have 2 versions of atlas, ABI incompatible. The point was to help for transition from g77 to gfortran ABI. Intrepid does not have this problem (they even went as far as removing g77 from the archives !). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Failing to build numpy properly on Ubuntu Hardy
On Thu, Dec 11, 2008 at 01:07:51PM +0900, David Cournapeau wrote: Gael Varoquaux wrote: I must have at some point installed the gfortran libraries by mistake. I was taken by surprise because I didn't expect Ubuntu to have 2 versions of atlas, ABI incompatible. The point was to help for transition from g77 to gfortran ABI. Intrepid does not have this problem (they even went as far as removing g77 from the archives !). Sure, I can understand that. I am on intrepid on half of my boxes so far. But not this one :). Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Failing to build numpy properly on Ubuntu Hardy
On Thu, Dec 11, 2008 at 06:10, Gael Varoquaux [EMAIL PROTECTED] wrote: Hi all, Looks like I am following the long line of people failing to build numpy :). I must admit I am clueless with building problems. Numpy builds alright, but I get: ImportError: /usr/lib/sse2/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done On import. This used to work a while ago. I am not sure what I changed, but it sure does fail. I really don't understand where the gfortran comes in. I tried building numpy with or without gfortran. From what I gather it is the numpy is being built by a different compiler than the atlas libraries (hurray for ABI compatibility), but I don't really understand how this is possible. How can I debug this? Cheers, Gaël I have encountered with such problem before. My solution is recompile the problem package(maybe atlas in your case) with -ff2c option passed to gfortran. -- Cheers, Grissiom ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Failing to build numpy properly on Ubuntu Hardy
Grissiom wrote: On Thu, Dec 11, 2008 at 06:10, Gael Varoquaux [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Hi all, Looks like I am following the long line of people failing to build numpy :). I must admit I am clueless with building problems. Numpy builds alright, but I get: ImportError: /usr/lib/sse2/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done On import. This used to work a while ago. I am not sure what I changed, but it sure does fail. I really don't understand where the gfortran comes in. I tried building numpy with or without gfortran. From what I gather it is the numpy is being built by a different compiler than the atlas libraries (hurray for ABI compatibility), but I don't really understand how this is possible. How can I debug this? Cheers, Gaël I have encountered with such problem before. My solution is recompile the problem package(maybe atlas in your case) with -ff2c option passed to gfortran. This is a bad idea: it won't work with libraries which are not built with this option, and the error won't always be easy to detect (one key difference is that wo ff2c, complex variables are passed by value by gfortran, whereas they are passed by reference with the ff2c option - which means crash and/or corruption). http://wiki.debian.org/GfortranTransition The only viable solution is to avoid mixing g77-built and gfortran-built libraries (there is now a simple test which tries to detect those mix in both numpy and scipy), cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Failing to build numpy properly on Ubuntu Hardy
On Thu, Dec 11, 2008 at 15:13, David Cournapeau [EMAIL PROTECTED] wrote: Grissiom wrote: I have encountered with such problem before. My solution is recompile the problem package(maybe atlas in your case) with -ff2c option passed to gfortran. This is a bad idea: it won't work with libraries which are not built with this option, and the error won't always be easy to detect (one key difference is that wo ff2c, complex variables are passed by value by gfortran, whereas they are passed by reference with the ff2c option - which means crash and/or corruption). http://wiki.debian.org/GfortranTransition The only viable solution is to avoid mixing g77-built and gfortran-built libraries (there is now a simple test which tries to detect those mix in both numpy and scipy), cheers, David Thanks for pointing out my mistake ;) -- Cheers, Grissiom ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
