Re: [Numpy-discussion] NumPy SVN broken
On Thu, Oct 8, 2009 at 11:39 AM, Travis Oliphant oliph...@enthought.com wrote: I apologize for the mis communication that has occurred here. No problem I did not understand that there was a desire to keep ABI compatibility with NumPy 1.3 when NumPy 1.4 was released. The datetime merge was made under that presumption. I had assumed that people would be fine with recompilation of extension modules that depend on the NumPy C-API. There are several things that needed to be done to merge in new fundamental data-types. Why don't we call the next release NumPy 2.0 if that helps things? Personally, I'd prefer that over hacks to keep ABI compatibility. Keeping ABI compatibility by itself is not an hack - the current workaround is an hack, but that's only because the current way of doing things in code generator is a bit ugly, and I did not want to spend too much time on it. It is purely an implementation issue, the fundamental idea is straightforward. If you want a cleaner solution, I can work on it. I think the hour or so that it would take is worth it compared to breaking many people's code. It feels like we are working very hard to track ABI issues that can also be handled with dependency checking and good package management. I think ABI issues are mostly orthogonal to versioning - generally, versions are related to API changes (API changes is what should drive ABI changes, at least for projects like numpy). I would prefer passing to numpy 2.0 when we really need to break ABI and API - at that point, I think we should also think hard about changing our structures and all to make them more robust to those changes (using pimp-like strategies in particular). David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] robustness strategies
Alan G Isaac wrote: On 10/7/2009 10:51 PM, David Cournapeau wrote: pimp-like strategies Which means ... ? The idea is to put one pointer in you struct instead of all members - it is a form of encapsulation, and it is enforced at compile time. I think part of the problem with changing API/ABI in numpy is that the headers show way too much information. I would really like to improve this, but this would clearly break the ABI (and API - a lot of macros would have to go). There is a performance cost of one more indirection (if you have a pointer to a struct, you need to dereference both the struct and the D pointer inside), but for most purpose, that's likely to be negligeable, except for a few special cases (like iterators). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy SVN broken
On Wed, Oct 7, 2009 at 2:04 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 6, 2009 at 10:50 AM, David Cournapeau courn...@gmail.com wrote: On Wed, Oct 7, 2009 at 1:36 AM, Charles R Harris charlesr.har...@gmail.com wrote: 2009/10/6 Stéfan van der Walt ste...@sun.ac.za Hi all, The current SVN HEAD of NumPy is broken and should not be used. Extensions compiled against this version may (will) segfault. Can you be more specific? I haven't had any problems running current svn with scipy. The version itself is fine, but the ABI has been changed in an incompatible way: if you have an extension built against say numpy 1.2.1, and then use a numpy built from sources after the datetime merge, it will segfault right away. It does so for scipy and several custom extensions. The abi breakage was found to be the datetime merge. Ah... That's a fine kettle of fish. Any idea what ABI calls are causing the problem? Maybe the dtype change wasn't made in a compatible way. IIRC, something was added to the dtype? Yes, but that should not cause trouble. Adding members to structure should be fine. I quickly look at the diff, and some changes in the code generators look suspicious, e.g.: types = ['Generic','Number','Integer','SignedInteger','UnsignedInteger', - 'Inexact', + 'Inexact', 'TimeInteger', 'Floating', 'ComplexFloating', 'Flexible', 'Character', 'Byte','Short','Int', 'Long', 'LongLong', 'UByte', 'UShort', 'UInt', 'ULong', 'ULongLong', 'Float', 'Double', 'LongDouble', 'CFloat', 'CDouble', 'CLongDouble', 'Object', 'String', 'Unicode', - 'Void'] + 'Void', 'Datetime', 'Timedelta'] As the list is used to initialize some values from the API function pointer array, inserts should be avoided. You can see the consequence on the generated files, e.g. part of __multiarray_api.h diff between datetimemerge and just before: #define PyFloatingArrType_Type (*(PyTypeObject *)PyArray_API[16]) #define PyComplexFloatingArrType_Type (*(PyTypeObject *)PyArray_API[17]) #define PyFlexibleArrType_Type (*(PyTypeObject *)PyArray_API[18]) #define PyCharacterArrType_Type (*(PyTypeObject *)PyArray_API[19]) #define PyByteArrType_Type (*(PyTypeObject *)PyArray_API[20]) #define PyShortArrType_Type (*(PyTypeObject *)PyArray_API[21]) #define PyIntArrType_Type (*(PyTypeObject *)PyArray_API[22]) #define PyLongArrType_Type (*(PyTypeObject *)PyArray_API[23]) #define PyLongLongArrType_Type (*(PyTypeObject *)PyArray_API[24]) #define PyUByteArrType_Type (*(PyTypeObject *)PyArray_API[25]) #define PyUShortArrType_Type (*(PyTypeObject *)PyArray_API[26]) #define PyUIntArrType_Type (*(PyTypeObject *)PyArray_API[27]) #define PyULongArrType_Type (*(PyTypeObject *)PyArray_API[28]) #define PyULongLongArrType_Type (*(PyTypeObject *)PyArray_API[29]) #define PyFloatArrType_Type (*(PyTypeObject *)PyArray_API[30]) #define PyDoubleArrType_Type (*(PyTypeObject *)PyArray_API[31]) #define PyLongDoubleArrType_Type (*(PyTypeObject *)PyArray_API[32]) #define PyCFloatArrType_Type (*(PyTypeObject *)PyArray_API[33]) #define PyCDoubleArrType_Type (*(PyTypeObject *)PyArray_API[34]) #define PyCLongDoubleArrType_Type (*(PyTypeObject *)PyArray_API[35]) #define PyObjectArrType_Type (*(PyTypeObject *)PyArray_API[36]) #define PyStringArrType_Type (*(PyTypeObject *)PyArray_API[37]) #define PyUnicodeArrType_Type (*(PyTypeObject *)PyArray_API[38]) #define PyVoidArrType_Type (*(PyTypeObject *)PyArray_API[39]) --- #define PyTimeIntegerArrType_Type (*(PyTypeObject *)PyArray_API[16]) #define PyFloatingArrType_Type (*(PyTypeObject *)PyArray_API[17]) #define PyComplexFloatingArrType_Type (*(PyTypeObject *)PyArray_API[18]) #define PyFlexibleArrType_Type (*(PyTypeObject *)PyArray_API[19]) #define PyCharacterArrType_Type (*(PyTypeObject *)PyArray_API[20]) #define PyByteArrType_Type (*(PyTypeObject *)PyArray_API[21]) #define PyShortArrType_Type (*(PyTypeObject *)PyArray_API[22]) #define PyIntArrType_Type (*(PyTypeObject *)PyArray_API[23]) #define PyLongArrType_Type (*(PyTypeObject *)PyArray_API[24]) #define PyLongLongArrType_Type (*(PyTypeObject *)PyArray_API[25]) #define PyUByteArrType_Type (*(PyTypeObject *)PyArray_API[26]) #define PyUShortArrType_Type (*(PyTypeObject *)PyArray_API[27]) #define PyUIntArrType_Type (*(PyTypeObject *)PyArray_API[28]) #define PyULongArrType_Type (*(PyTypeObject *)PyArray_API[29]) #define PyULongLongArrType_Type (*(PyTypeObject *)PyArray_API[30]) #define PyFloatArrType_Type (*(PyTypeObject *)PyArray_API[31]) #define PyDoubleArrType_Type (*(PyTypeObject *)PyArray_API[32]) #define PyLongDoubleArrType_Type (*(PyTypeObject *)PyArray_API[33]) #define PyCFloatArrType_Type (*(PyTypeObject *)PyArray_API[34]) #define PyCDoubleArrType_Type (*(PyTypeObject *)PyArray_API[35]) #define PyCLongDoubleArrType_Type (*(PyTypeObject *)PyArray_API[36]) #define PyObjectArrType_Type (*(PyTypeObject *)PyArray_API[37
Re: [Numpy-discussion] Windows 64-bit
On Fri, Oct 2, 2009 at 3:43 AM, Klaus Noekel klaus.noe...@gmx.de wrote: - We need only numpy, not scipy. Does that imply that we have a good chance of producing an install ourselves with the current sources? The current sources can be compiled by visual studio in 64 bits mode without problem and should be quite stsable- you won't have a fast blas/lapack, though, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unpacking bytes directly in numpy
On Sat, Sep 26, 2009 at 10:33 PM, Thomas Robitaille
thomas.robitai...@gmail.com wrote:
Hi,
To convert some bytes to e.g. a 32-bit int, I can do
bytes = f.read(4)
i = struct.unpack('i', bytes)[0]
and the convert it to np.int32 with
i = np.int32(i)
However, is there a more direct way of directly transforming bytes
into a np.int32 type without the intermediate 'struct.unpack' step?
Assuming you have an array of bytes, you could just use view:
# x is an array of bytes, whose length is a multiple of 4
x.view(np.int32)
cheers,
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] datetime functionality: questions and requests
Hi there, Hi Travis, I have started looking at the new datetime code, with the idea that we should soon fix (for real) a release date for numpy 1.4.0. I have a few requests and questions: - since npy_datetime is conditionally defined to be 64 bits, why not typedefing it to npy_int64 ? - there are a few issues w.r.t int/long/npy_datetime used interchangeably. That's bound to cause trouble on 64 bits architectures. - For 1.4.0, there needs to be some minimal documentation with some examples + a few unit tests. In particular, I would like some unit tests which check for 64 bits issues. I was thinking about pushing for 1.4.0 in november, which means that those changes should be hopefully included within mid-october. Is this possible ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.any and np.all short-circuiting
On Fri, Sep 25, 2009 at 6:50 PM, Citi, Luca lc...@essex.ac.uk wrote: Thanks for your reply. So, what is the correct way to test a numpy development version without installing it in /usr/lib/... or /usr/local/lib/.. ? What do you guys do? Build in place, but test from outside the source tree. I for example have a makefile which does the build + test dance, but anything could do. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.any and np.all short-circuiting
On Fri, Sep 25, 2009 at 9:50 AM, Citi, Luca lc...@essex.ac.uk wrote: I am sorry. I followed your suggestion. I re-checked out the svn folder and then compiled with $ python setup.py build_src --inplace build_ext --inplace but I get the same behaviour. If I am inside I get the NameError, if I am outside and use path.insert, it successfully performs zero tests. There is a problem with numpy tests when you import numpy with the install being in the current directory. It happens when you build in place and launch python in the source directory, but it also happens if you happen to be in site-packages, and import numpy installed there. When the issue was last discussed, Robert suggested it was a nose issue. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy depends on OpenSSL ???
On Thu, Sep 24, 2009 at 1:20 AM, Charles R Harris charlesr.har...@gmail.com wrote: In any case, we should find a fix. I don't think we do - we requires a standard python install, and a python without hashlib is crippled. If you can't build python without openssl, I would consider this a python bug. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
Xavier Gnata wrote: Hi, I have a large 2D numpy array as input and a 1D array as output. In between, I would like to use C code. C is requirement because it has to be fast and because the algorithm cannot be written in a numpy oriented way :( (no way...really). Which tool should I use to achieve that? waeve.inline? pyrex? What is the baseline? That's only a data point, but I almost always use cython in those cases, unless I need 'very advanced' features of the C API in which case I just do it manually. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
On Mon, Sep 21, 2009 at 8:59 PM, Romain Brette romain.bre...@ens.fr wrote: David Warde-Farley a écrit : On 20-Sep-09, at 2:17 PM, Romain Brette wrote: Would anyone have thoughts about what the best hardware would be for Numpy? In particular, I am wondering about Intel Core i7 vs Xeon. Also, I feel that the limiting factor might be memory speed and cache rather than processor speed. What do you think? So, there are several different chips that bear the Xeon brand, you'd have to look at individual benchmarks. But if you're concerned about linear algebra performance, I'd say to go with the desktop version and spend some of the money you save on a license for the Intel Math Kernel Library to link NumPy against: http://software.intel.com/en-us/intel-mkl/ David Interesting, I might try Intel MKL. I use mostly element-wise operations (e.g. exp(x) or xx0, where x is a vector), do you think it would make a big difference? It won't make any difference for most operations, at least by default, as we only support the MKL for BLAS/LAPACK. IF the MKL gives a C99 interface to the math library, it may be possible to tweak the build process such as to benefit from them. Concerning the hardware, I have just bought a core i7 (the cheapest model is ~ 200$ now, with 4 cores and 8 Mb of shared cache), and the thing flies for floating point computation. My last computer was a pentium 4 so I don't have a lot of reference, but you can compute ~ 300e6 exp (assuming a contiguous array), and ATLAS 3.8.3 built on it is extremely fast - using the threaded version, the asymptotic peak performances are quite impressive. It takes for example 14s to inverse a 5000x5000 matrix of double. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
On Tue, Sep 22, 2009 at 12:57 AM, Neal Becker ndbeck...@gmail.com wrote: David Cournapeau wrote: On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. We don't want to create arrays of fixed-pt objects. That would be very wasteful. Maybe, but that would be a good way to prototype the thing. What I have in mind is that integer_bits, frac_bits are attributes of the entire arrays, not the individual elements. The array elements are just plain integers. That's not really how numpy arrays are designed: type-specific info should be in the dtype, not the array class. As Robert mentioned, the recently added datetime dtype shows an example on how to do it. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error while running installer
Gheorghe Postelnicu wrote: Hi guys, I just tried to run the 1.3.0 superpack installer and I get the following message box: Executing numpy installer failed. The details show the following lines: Output folder: C:\DOCUME~1\Ghighi\LOCALS~1\Temp Install dir for actual installers is C:\DOCUME~1\Ghighi\LOCALS~1\Temp Target CPU handles SSE2 Target CPU handles SSE3 native install (arch value: native) Install SSE 3 Extract: numpy-1.3.0-sse3.exe... 100% Execute: C:\DOCUME~1\Ghighi\LOCALS~1\Temp\numpy-1.3.0-sse3.exe Completed I just re-installed my Windows box. Are there any pre-requisites other than Python 2.6? I have installed Python 2.6.2. which version of windows are you using ? Is numpy installed (i.e. is the error message bogus or the installer actually failed ?) David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
Charles R Harris wrote: Hi René, On Fri, Sep 18, 2009 at 6:01 AM, René Dudfield ren...@gmail.com mailto:ren...@gmail.com wrote: Hello, as a big numpy user, and someone wanting to help with the python 3 migration, I'd like to help with a python 3.1 port of numpy. We(at the pygame project) have mostly completed our port of pygame to python 3.0 and 3.1 so can offer some insight into what it takes with a CPython extension. pygame supports python 2.3 through to 3.1, so it should be possible to also keep backwards compatibility with the port for numpy. We can also use some of the helper code we used for the pygame port. I haven't asked the other pygame developers if they are interested in helping too... but maybe they will be interested in helping too(since they use numpy too). I'd also be interested in other people helping with the effort. Once I have put some of the ground work in place, and got a few parts done it should be easier for other people to see what needs changing. If the python 3 port is something you'd want included with numpy, then I'd like to begin this weekend. I'm not super familiar with the numpy code base yet, so I'd like to do it in small changes making small parts compatible with py3k and then having them reviewed/tested. I can either start a branch myself using mecurial, or perhaps you'd like me to do the work somewhere else ( like in a branch in the numpy svn?). Which code should I base the port off? trunk ? Darren Dale and I are just getting started on a port and welcome any help you can offer. Because of the difficulty of maintaining two branches the only route that looks good at this point is to get the python parts of numpy in a state that will allow 2to3 to work and use #ifdef's in the c code. What was your experience with pygames? Because the numpy c code is difficult to come to grips with the easiest part for inexperienced c coders and newbies is to start on is probably the python code. There is a fair amount of that so some plan of attack and a check list probably needs to be set up. Any such list should be kept in svn along with any helpful notes about problems/solutions encountered along the way. We also need David C. to commit his build changes for py3k so we can actually build the whole thing when the time comes. (Hint, hint). I will try to update it, but I don't have much time ATM to work on python 3 issues - one issue is that we would need some kind of infrastructure so that incompatible distutils changes would be detected automatically (with a buildbot or something). Otherwise, since nobody can use python 3 and numpy ATM, the code will quickly bitrot. Concerning python version, I think 3.0 should not even be considered. It is already considered obsolete, and we can be fairly confident that there are 0 numpy users under python 3.0 :) cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
René Dudfield wrote: ah, oh well. I'll just upload diffs as I go. Which versions of python is numpy to support? 2.4 and above, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] matlab for numpy users
Hi Christian, Christian K. wrote: Hi, this is probaby an unusual question here from someone used to numpy who is forced to work with matlab and it is not exactly the right place to ask. Sorry for that. Is there something like broadcasting in matlab? Not really (except for trivial things like scalar-matrix operations). The usual way to do it in matlab is repmat, which helps you doing 'manual broadcasting'. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Moved matrix class into separate module
Hi, I just wanted to mention I integrated a patch from some time ago to make numpy.core independent from other numpy modules. This is really useful when working on involved changes at the C level. This meant moving some stuff around, in particular the matrix class and utilities is now into numpy.matrixclass module. The numpy namespace is of course unchanged. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moved matrix class into separate module
On Thu, Sep 17, 2009 at 12:32 AM, Christopher Hanley chan...@stsci.edu wrote: Hi, When I try running the tests on a fresh build from the trunk I receive 28 errors. Most of the errors are of the form: NameError: global name 'matrix' is not defined It looks like there was some change to the numpy namespace. I can provide a full listing of the unit test errors if desired. Yes please - I do not see those errors on both mac os x and linux. You should also make sure that your svn checkout, build and install directories are not polluted by old cruft. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit Fedora 9 a=numpy.zeros(0x80000000, dtype='b1')
Charles R Harris wrote: On Sat, Sep 12, 2009 at 9:03 AM, Citi, Luca lc...@essex.ac.uk mailto:lc...@essex.ac.uk wrote: I just realized that Sebastian posted its 'uname -a' and he has a 64bit machine. In this case it should work as mine (the 64bit one) does. Maybe during the compilation some flags prevented a full 64bit code to be compiled? __ Ints are still 32 bits on 64 bit machines, but the real question is how python interprets the hex value. That's not a python problem: the conversion of the object to a C int/long happens in numpy (in PyArray_IntpFromSequence in this case). I am not sure I understand exactly what the code is doing, though. I don't understand the rationale for #ifdef/#endif in the one item in shape tuple case (line 521 and below), as well as the call to PyNumber_Int, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dot product performance on python 2.6 (windows)
V. Armando Solé wrote: Hello, I have found performance problems under windows when using python 2.6 In my case, they seem to be related to the dot product. The following simple script: import numpy import time a=numpy.arange(100.) a.shape=1000,1000 t0=time.time() b=numpy.dot(a.T,a) print Elapsed time = ,time.time()-t0 reports an Elapsed time of 1.4 seconds under python 2.5 and 15 seconds under python 2.6 Same version of numpy, same machine, official numpy installers for windows (both with nosse flag) Could you confirm this by pasting the output of numpy.show_config() in both versions ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dot product performance on python 2.6 (windows)
V. Armando Solé wrote: Hello, It seems to point towards a packaging problem. In python 2.5, I can do: import numpy.core._dotblas as dotblas dotblas.__file__ and I get: C:\\Python25\\lib\\site-packages\\numpy\\core\\_dotblas.pyd That's where the error lies: if you install with nosse, you should not get _dotblas.pyd at all. The 15 second is the 'normal' speed if you don't use ATLAS. I will look into the packaging problem - could you open an issue on numpy trac, so that I don't forget about it ? cheers, David In python 2.6: import numpy.core._dotblas as dotblas ... ImportError: No module named _dotblas and, of course, I cannot find the _dotblas.pyd file in the relevant directories. Best regards, Armando ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dot product performance on python 2.6 (windows)
V. Armando Solé wrote: David Cournapeau wrote: V. Armando Solé wrote: Hello, It seems to point towards a packaging problem. In python 2.5, I can do: import numpy.core._dotblas as dotblas dotblas.__file__ and I get: C:\\Python25\\lib\\site-packages\\numpy\\core\\_dotblas.pyd That's where the error lies: if you install with nosse, you should not get _dotblas.pyd at all. Why? The nosse for python 2.5 has _dotblas.pyd Yes, and it should not - because the _dotblas.pyd uses SSE2 instructions. The python 2.6 installer is the correct one, python 2.5 is not. Is it impossible to get it compiled under python 2.6 without using sse2 or sse3? It is possible to compile anything you want if you are willing to go through the hassle of compiling ATLAS on windows. The binary installer only uses ATLAS (and hence build _dotblas) for SSE2 and SSE3. The low availability of machines with only SSE does not worth the hassle to do it anymore (but again, that's only an issue of the official binaries, you can still compile your own atlas). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Huge arrays
Kim Hansen wrote: On 9-Sep-09, at 4:48 AM, Francesc Alted wrote: Yes, this later is supported in PyTables as long as the underlying filesystem supports files 2 GB, which is very usual in modern operating systems. I think the OP said he was on Win32, in which case it should be noted: FAT32 has its upper file size limit at 4GB (minus one byte), so storing both your arrays as one file on a FAT32 partition is a no-no. David Strange, I work on Win32 systems, and there I have no problems storing data files up to 600 GB (have not tried larger) in size stored on RAID0 disk systems of 2x1TB, I can also open them and seek in them using Python. It is a FAT32 limitation, not a windows limitation. NTFS should handle large files without much trouble, and I believe the vast majority of windows installations (= windows xp) use NTFS and not FAT32. I certainly have not seen a windows installed on FAT32 for a very long time. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Huge arrays
On Wed, Sep 9, 2009 at 9:30 AM, Daniel Platzmail.to.daniel.pl...@googlemail.com wrote: Hi, I have a numpy newbie question. I want to store a huge amount of data in an array. This data come from a measurement setup and I want to write them to disk later since there is nearly no time for this during the measurement. To put some numbers up: I have 2*256*200 int16 numbers which I want to store. I tried data1 = numpy.zeros((256,200),dtype=int16) data2 = numpy.zeros((256,200),dtype=int16) This works for the first array data1. However, it returns with a memory error for array data2. I have read somewhere that there is a 2GB limit for numpy arrays on a 32 bit machine This has nothing to do with numpy per se - that's the fundamental limitation of 32 bits architectures. Each of your array is 1024 Mb, so you won't be able to create two of them. The 2Gb limit is a theoretical upper limit, and in practice, it will always be lower, if only because python itself needs some memory. There is also the memory fragmentation problem, which means allocating one contiguous, almost 2Gb segment will be difficult. If someone has an idea to help me I would be very glad. If you really need to deal with arrays that big, you should move on 64 bits architecture. That's exactly the problem they are solving. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] question about future support for python-3
On Wed, Sep 9, 2009 at 9:37 AM, Darren Daledsdal...@gmail.com wrote: Hi David, I already gave my own opinion on py3k, which can be summarized as: - it is a huge effort, and no core numpy/scipy developer has expressed the urge to transition to py3k, since py3k does not bring much for scientific computing. - very few packages with a significant portion of C have been ported to my knowledge, hence very little experience on how to do it. AFAIK, only small packages have been ported. Even big, pure python projects have not been ported. The only big C project to have been ported is python itself, and it broke compatibility and used a different source tree than python 2. - it remains to be seen whether we can do the py3k support in the same source tree as the one use for python = 2.4. Having two source trees would make the effort even much bigger, well over the current developers capacity IMHO. The only area where I could see the PSF helping is the point 2: more documentation, more stories about 2-3 transition. I'm surprised to hear you say that. I would think additional developer and/or financial resources would be useful, for all of the reasons you listed. If there was enough resources to pay someone very familiar with numpy codebase for a long time, then yes, it could be useful - but I assume that's out of the question. This would be very expensive as it would requires several full months IMO. The PSF could help for the point 3, by porting other projects to py3k and documenting it. The only example I know so far is pycog2 (http://mail.python.org/pipermail/python-porting/2008-December/10.html). Paying people to do documentation about porting C code seems like a good way to spend money: it would be useful outside numpy community, and would presumably be less costly. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] question about future support for python-3
On Wed, Sep 9, 2009 at 4:21 AM, Darren Daledsdal...@gmail.com wrote: I'm not a core numpy developer and don't want to step on anybody's toes here. But I was wondering if anyone had considered approaching the Python Software Foundation about support to help get numpy working with python-3? I already gave my own opinion on py3k, which can be summarized as: - it is a huge effort, and no core numpy/scipy developer has expressed the urge to transition to py3k, since py3k does not bring much for scientific computing. - very few packages with a significant portion of C have been ported to my knowledge, hence very little experience on how to do it. AFAIK, only small packages have been ported. Even big, pure python projects have not been ported. The only big C project to have been ported is python itself, and it broke compatibility and used a different source tree than python 2. - it remains to be seen whether we can do the py3k support in the same source tree as the one use for python = 2.4. Having two source trees would make the effort even much bigger, well over the current developers capacity IMHO. The only area where I could see the PSF helping is the point 2: more documentation, more stories about 2-3 transition. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Huge arrays
On Wed, Sep 9, 2009 at 2:10 PM, Sebastian Haaseseb.ha...@gmail.com wrote: Hi, you can probably use PyTables for this. Even though it's meant to save/load data to/from disk (in HDF5 format) as far as I understand, it can be used to make your task solvable - even on a 32bit system !! It's free (pytables.org) -- so maybe you can try it out and tell me if I'm right You still would not be able to load a numpy array 2 Gb. Numpy memory model needs one contiguously addressable chunk of memory for the data, which is limited under the 32 bits archs. This cannot be overcome in any way AFAIK. You may be able to save data 2 Gb, by appending several chunks 2 Gb to disk - maybe pytables supports this if it has large file support (which enables to write files 2Gb on a 32 bits system). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] datetime-related import slowdown
Hi, I noticed that numpy import times significantly are significantly worse than it used to be, and those are related to recent datetime related changes: # One month ago time python -c import numpy - 141ms # Now: time python -c import numpy - 202ms Using bzr import profiler, most of the slowdown comes from mx_datetime, and I guess all the regex compilation within (each re.compile takes several ms each in there). Is there a way to make this faster, at least for people not using datetime ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/scipy/matplotlib + 10.6 + Apple python 2.6.1
On Mon, Sep 7, 2009 at 6:00 PM, George Nursergnur...@googlemail.com wrote: There are some interesting instructions on how to make this work at http://blog.hyperjeff.net/?p=160. However I'm not sure that the recommendation to rename the Apple-supplied version of numpy is consistent with previous advice I've seen on this mailing list. I think it is best to avoid touching anything in /System. The better solution is to install things locally, at least if you don't need to share with several users one install. With python 2.6, there is actually a very simple way to do so, using the --user option of python. You install everything using the --user option, and this will get installed into your $HOME, and python will look there first. Also, FFTW is not needed by scipy anymore. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] mixing -fno-exceptions with swig c++ wrappers to python
Rohit Garg wrote: Yeah, that's what I meant. If my code does not use exceptions, then is it safe to use -fno-exceptions? You would have to look at g++ documentation - but if it is safe for your code, numpy should not make it unsafe. I am not sure what not using exception means in C++, though, as new throws exception for example. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] mixing -fno-exceptions with swig c++ wrappers to python
Rohit Garg wrote: Hi, I am using swig to expose a c++ class to Python. I am wondering if it is safe to use the -fno-exceptions option while compiling the wrappers. I am also using the typemaps present in the numpy.i file that comes with numpy. It will mostly depend on the code you are wrapping and your toolchain. It should not cause trouble w.r.t numpy, as numpy does not use C++ at all. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fwrite() failure in PyArray_ToFile
On Sat, Sep 5, 2009 at 2:01 PM, Charles R Harrischarlesr.har...@gmail.com wrote: On Fri, Sep 4, 2009 at 10:29 PM, Sturla Molden stu...@molden.no wrote: Charles R Harris skrev: The size of long depends on the compiler as well as the operating system. On linux x86_64, IIRC, it is 64 bits, on Windows64 I believe it is 32. Ints always seem to be 32 bits. If I remember the C standard correctly, a long is guaranteed to be at least 32 bits, whereas an int is guaranteed to be at least 16 bits. A short is at least 16 bits and a long long is 64 bits. Then there is intptr_t which is wide enough to store a pointer, but could be wider. A size_t usually has the size of a pointer but not always (on segment and offset architectures they might differ). Considering PEP353, should we be indexing with Py_ssize_t instead of npy_intp? I believe (correct me if I'm wrong) npy_intp is typedef'ed to Py_intptr_t. The problem is that Py_ssize_t showed up in Python 2.5 and we support 2.4. For earlier versions folks usually typedef Py_ssize_t to int, which works for python compatibility but would cause problems for us if we used it as the npy_intp type. We could use our own npy_ssize_t, though. I find npy_intp for indexing quite confusing. I have started a patch toward this during the scipy09 sprints. There are a lot of bogus loops in numpy and scipy where the indexing loop is an int BTW. For the formatting, we should have our own formatting macro to deal with printing those - I have meant to implement them, but it is a bit of work to do it correctly as a printf format for size_t only appeared in C99 (the usual workaround to use usigned long is wrong on windows 64). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] snow leopard issues with numpy
Chad Netzer wrote: On Thu, Sep 3, 2009 at 4:02 PM, Wolfgang Kerzendorfwkerzend...@googlemail.com wrote: my version of python is the one that comes with snow leopard: 2.6.1 hope that helps Huh? I upgraded to Snow Leopard over my Leopard system (i.e not a fresh install), and my default is python2.5: $ python Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Hmmm, also that looks very old... [snooping] Okay, so it seems I probably have an old Mac Python installed in /Library/Frameworks, that perhaps was overriding the new Snow Leopard python? $ type -a python python is /Library/Frameworks/Python.framework/Versions/Current/bin/python python is /usr/bin/python $ ls -l /Library/Frameworks/Python.framework/Versions total 8 drwxr-xr-x 9 root admin 306 Sep 5 2006 2.4/ drwxrwxr-x 10 root admin 340 Dec 11 2006 2.5/ lrwxr-xr-x 1 root admin3 Apr 5 16:53 Current@ - 2.5 Geez... I think I was using the MacPorts python, which I deleted after installing SnowLeopard, and that exposed some old still installed frameworks. $ sudo rm /Library/Frameworks/Python.framework/Versions/Current/bin/python $ type -a python python is /usr/bin/python $ hash python $ python Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type help, copyright, credits or license for more information. Yep, got rid of the old framework executable, and now it finds the system python executable. Man, there is a lot of old cruft of various sort in my /Library/Frameworks. So, note to those upgrading to Snow Leopard, look in your /Library?Frameworks folder for old just that may still be overriding the newer /System/Library/Frameworks stuff: Concerning python on snow Leopard, two things will cause quite a few issues and I expect a few emails on the ML in the next few weeks :): - the system python is now 64 bits - gcc now targets x86_64 by default (i.e. without -arch). Apple seems to push 64 bits quite strongly in Snow Leopard, with almost every app being 64 bits on my system except the kernel of course. OTOH, this should make numpy/scipy more easily available for 64 bits on mac os x, assuming the changes to python itself will be integrated upstream (I doubt Apple made an effort to send patches upstream themselves). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] snow leopard issues with numpy
On Fri, Sep 4, 2009 at 1:49 AM, Charles R Harrischarlesr.har...@gmail.com wrote: On Thu, Sep 3, 2009 at 10:39 AM, Robert Kern robert.k...@gmail.com wrote: On Thu, Sep 3, 2009 at 11:13, Charles R Harrischarlesr.har...@gmail.com wrote: On Thu, Sep 3, 2009 at 10:02 AM, Wolfgang Kerzendorf wkerzend...@googlemail.com wrote: I just installed numpy and scipy (both svn) on OS X 10.6 and just got scipy to work with Robert Kern's help. Playing around with numpy I got the following segfault: http://pastebin.com/m35220dbf I hope someone can make sense of it. Thanks in advance Wolfgang I'm going to guess it is a python problem. Which version of python do you have and where did it come from? Or a matplotlib problem. _path.so is theirs. Likely, then. I had to recompile matplotlib from svn when I upgraded to Fedora 11. Looks like C++ code, and snow leopard has gcc 4.2, which is likely to have some subtle ABI incompatibilities with 4.0 (the one from leopoard). So yes, a matplotlib rebuild would be the first thing to try. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] future directions
On Fri, Aug 28, 2009 at 11:18 AM, Robert Kernrobert.k...@gmail.com wrote: On Fri, Aug 28, 2009 at 09:15, Christopher Barkerchris.bar...@noaa.gov wrote: Joe Harrington wrote: However, there are two natural forklets coming up. The first is Python 3.0, which will necessitate some API changes. Absolutely! This seems like a no-brainer. I don't think we are talking about really major changes to the numpy API anyway, generally clean-up, and there is no way anyone is going to get their Py2 code working on Py3 without tweaking it anyway, this is the time to do it. No, it is the *worst* time to do it. We have been asked by the Python developer team *not* to use the Python 3 transition to break all kinds of other backwards compatibility. AFAIK, the main argument is that this would allow for easier transition, since someone could use 2to3 to make the transition from numpy for python 2 to numpy for python 3. But is it even possible for a large package like numpy ? I don't see how the C api for example could be backward compatible, since the API with PyString and PyInt would have to be changed. I guess time will tell, once other package with a lof of C will be converted. I am curious to see whether this advice from the python dev community will be followed at all. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Quick question on NumPy builds vs. Cython
On Tue, Aug 25, 2009 at 9:44 PM, Charles R Harrischarlesr.har...@gmail.com wrote: On Tue, Aug 25, 2009 at 11:30 AM, David Cournapeau courn...@gmail.com wrote: Hi Dag, On Tue, Aug 25, 2009 at 12:19 PM, Dag Sverre Seljebotnda...@student.matnat.uio.no wrote: [Let me know if this should go to numpy-discuss instead.] I guess this can be discussed on the ML as well (I CC to the list). I see that there are currently two modes, and that it is possible to build NumPy using a master .c-file #include-ing the rest. (Which is much more difficult to support using Cython, though not impossible.) Is there any plans for the one-file build to go away, or is supporting this a requirement? This is a requirement, as supporting this depends on non standard compilers extensions (that's why it is not the default - but it works well, I am always using this mode when working on numpy since the build/test/debug cycle is so much shorter with numscons and this). The basic problem is as follows: - On Unix at least, a function is exported in a shared library by default. - The usual way to avoid polluting the namespace is to put static in front of it - You can't reuse a static function in another compilation unit (there is no friend static). So what happens in the multi-files build is that the function are tagged as hidden instead of static, with hidden being __attribute__((hidden)) for gcc, nothing for MS compiler (on windows, you have to tag the exported functions, nothing is exported by default), and will break on other platforms. Does the build actually break or is it the case that a lot of extraneous names become visible, increasing the module size and exposing functions that we don't what anyone to access? I think the latter. The number of exported symbols is pretty high, though. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Quick question on NumPy builds vs. Cython
Hi Dag, On Tue, Aug 25, 2009 at 12:19 PM, Dag Sverre Seljebotnda...@student.matnat.uio.no wrote: [Let me know if this should go to numpy-discuss instead.] I guess this can be discussed on the ML as well (I CC to the list). I see that there are currently two modes, and that it is possible to build NumPy using a master .c-file #include-ing the rest. (Which is much more difficult to support using Cython, though not impossible.) Is there any plans for the one-file build to go away, or is supporting this a requirement? This is a requirement, as supporting this depends on non standard compilers extensions (that's why it is not the default - but it works well, I am always using this mode when working on numpy since the build/test/debug cycle is so much shorter with numscons and this). The basic problem is as follows: - On Unix at least, a function is exported in a shared library by default. - The usual way to avoid polluting the namespace is to put static in front of it - You can't reuse a static function in another compilation unit (there is no friend static). So what happens in the multi-files build is that the function are tagged as hidden instead of static, with hidden being __attribute__((hidden)) for gcc, nothing for MS compiler (on windows, you have to tag the exported functions, nothing is exported by default), and will break on other platforms. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] Deprecate chararray [was Plea for help]
On Thu, Aug 20, 2009 at 10:32 AM, Christopher Hanleychan...@stsci.edu wrote: Another concern is that we told people coming from numarray to use this module. It is my opinion that at this point in the numpy release cycle that an API change needs a very strong justification. Anecdotes about the number of users, a change or die philosophy, and an un- articulated notion of the spirit of numpy do not in my consideration meet that high bar. I agree those are not strong reasons without more backing. What worries me the most, in both numpy and scipy, is code that nobody knows about, without any test or documentation. When it breaks, we can't fix it. That's unsustainable in the long term, because it takes a lot of time that people could spend somewhere else more useful. Especially when you have C code which does not work on some platforms, with new version of python (python 3k port, for example). I much prefer removing code to having code that barely works and cannot be maintained. Old code that people are ready to maintain, I have nothing against. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] nosetests and permissions
On Thu, Aug 20, 2009 at 2:06 PM, Chris Colbertsccolb...@gmail.com wrote: the issue is that the files are executable. I have no idea why they are set that way either. This is numpy 1.3.0 built from source. Which sources are you using ? The tarball on sourceforge, from svn, etc... ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] why does b[:-0] not work, and is there an elegant solution?
On Wed, Aug 19, 2009 at 5:50 AM, Neil Martinsen-Burrelln...@wartburg.edu wrote: On Aug 19, 2009, at 7:25 AM, Mark Bakker wrote: I compute the index of the last term in an array that I need and call the index n. I can then call the array b as b[:-n] If I need all terms in the array, the logical syntax would be: b[:-0] but that doesn't work. Any reason why that has not been implemented? Any elegant workaround? Because there is no negative zero as an integer: -0 == 0 True Not that it matters for the discussion, but -0.0 == 0.0: x = np.array(np.PZERO) y = np.array(np.NZERO) y == x # True 1 / x == 1 / y # False: inf and negative inf The only way to differentiate the number by itself is signbit, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ATLAS, NumPy and Threading
Peter Jeremy wrote: [Apologies if anyone sees this twice - the first copy appears to have disappeared into a black hole] Should ATLAS be built with or without threading support for use with NumPy? The NumPy documentation just says that ATLAS will be used if found but gives no indication of how ATLAS should be built. Both threaded and non threaded should be usable, at least on unix and mac os x. I don't know about the situation on windows, though. The FreeBSD notice in system_info may be obsolete, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures r7300
On Sat, Aug 8, 2009 at 9:38 PM, lukshun...@gmail.com wrote: Hi, I got 16 test failures after building r7300 from svn on debian/sid/i386. Seems all related to complex linear algebra modules. Are you using atlas ? (numpy.show_config() output) If so, did you compile it by yourself ? Did you compile everything with gfortran (do you have g77 installed). Problems related to complex are almost always caused by fortran compilers mismatch, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures r7300
On Sat, Aug 8, 2009 at 10:33 PM, lukshun...@gmail.com wrote: David Cournapeau wrote: On Sat, Aug 8, 2009 at 9:38 PM, lukshun...@gmail.com wrote: Hi, I got 16 test failures after building r7300 from svn on debian/sid/i386. Seems all related to complex linear algebra modules. Are you using atlas ? (numpy.show_config() output) Yes, it's libatlas-sse2 3.6.0-24 debian/sid package. I wonder if debian atlas package has the same problem as on recent Ubuntu. [DEFAULT] library_dirs = /usr/lib/sse2 [blas_opt] libraries = f77blas, cblas, atlas [lapack_opt] libraries = lapack_atlas, f77blas, cblas, atlas but it seems not to pick up the liblapack_atlas.so from the debian atlas package. there is no need to do this I think: it should work out of the box without any site.cfg (the point is that it would make it easier to change which atlas is loaded at runtime for further debugging of the issue). Does the numpy.show_config() show that the debian atlas libaries are compiled with f77? No, the f77 refers to the fortran 77 dialect, not the compiler. What I would try is first install libatlas-base (or whatever it is called on sid), i.e. the non sse version, and compare test output with both sse2/nosse (e.g. using LD_LIBRARY_PATH to point to /usr/lib so that the nosse is loaded, you can check using ldd which one is loaded by ld). David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures for rev7299
Christopher Hanley wrote: Hi, I receive the following test errors after building numpy rev7229 from svn: Yep, a bug slipped in the last commit, I am fixing it right now, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funded work on Numpy: proposed improvements and request for feedback
Bruce Southey wrote: So if 'C99-like' is going to be the near term future, is there any point in supporting non-C99 environments with this work? There may be a misunderstanding: if the platform support C99 complex, then we will use it, and otherwise, we will do as today, that is define our own type. The advantages of reusing the C99 complex type if available: - if yourself do not care about portability, you can use the numpy complex typedef as a C99 complex, using addition, division, etc... operators. - we can reuse the math library. I also need some sort of proper C99 support for windows 64 (more exactly to reimplement a minimal libgfortran buildable by MS compiler). That is, is the limitation in the compiler, operating system, processor or some combination of these? That's purely a compiler issue. Of course, the main culprit is MS compiler. MS explicitly stated they did not care about proper C support. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] GPU Numpy
Olivier Grisel wrote: OpenCL is definitely the way to go for a cross platform solution with both nvidia and AMD having released beta runtimes to their respective developer networks (free as in beer subscription required for the beta dowload pages). Final public releases to be expected around 2009 Q3. What's the status of opencl on windows ? Will MS have its own direct-x specific implementation ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funded work on Numpy: proposed improvements and request for feedback
Hi Chuck, Charles R Harris wrote: To make purely computational code available to third parties, two things are needed: 1. the code itself needs to make the split explicit. 2. there needs to be support so that reusing those functionalities is as painless as possible, from a build point of view (Note: this is almost done in the upcoming numpy 1.4.0 as long as static linking is OK). Ah, it itches. This is certainly a worthy goal, but are there third parties who have expressed an interest in this? I mean, besides trying to avoid duplicate bits of code in Scipy. Actually, I think that's what interests people around the Nipy project the most. In particular, they need to reuse lapack and random quite a bit, and for now, they just duplicate the code, with all the problems it brings (duplication, lack of reliability as far as cross platform is concerned, etc...). Splitting the code -- The amount of work is directly proportional to the amount of functions to be made available. The most obvious candidates are: 1. C99 math functions: a lot of this has already been done. In particular math constants, and special values support is already implemented. Almost every real function in numpy has a portable npy\_ implementation in C. 2. C99-like complex support: this naturally extends the previous series. The main difficult is to support platforms without C99 complex support, and the corresponding C99 complex functions. 3. FFT code: there is no support to reuse FFT at the C level at the moment. 4. Random: there is no support either 5. Linalg: idem. This is good. I think it should go along with code reorganization. The files are now broken up but I am not convinced that everything is yet where it should be. Oh, definitely agreed. Another thing I would like in that spirit is to split the numy headers like in Python itself: ndarrayobject.h would still pull out everything (for backward compatibility reasons), but people could only include a few headers if they want to. The rationale for me is when I work on numpy itself: it is kind of stupid that everytime I change the iterator structures, the whole numpy core has to be rebuilt. That's quite wasteful and frustrating. Another rationale is to be able to compile and test a very minimal core numpy (the array object + a few things). I don't see py3k port being possible in a foreseeable future without this. The complex support could be a major effort in its own right if we need to rewrite all the current functions. That said, it would be nice if the complex support was separated out like the current real support. Test to go along with it would be helpful. This also ties in with having build support for many platforms. Pauli has worked on this a little, and I have actually worked quite a bit myself because I need a minimal support for windows 64 bits support (to fake libgfortran). I have already implemented around 10 core complex functions (cabs, cangle, creal, cimag, cexp, cpow, csqrt, clog, ccos, csin, ctan), in such a way that native C99 complex are used on platforms which support it, and there is a quite thorough test suite which tests every special value condition (negative zero, inf, nan) as specified in the C99 standard. Still lacks actual values (!), FPU exception and branch cuts tests, and thorough tests on major platforms. And quite a few other functions would be useful (hyperbolic trigo). Build support - Once the code itself is split, there needs some support so that the code can be reused by third-parties. The following issues need to be solved: 1. Compile the computational code into shared or static library 2. Once built, making the libraries available to third parties (distutils issues). Ideally, it should work in installed, in-place builds, etc\.\.\. situations. 3. Versioning, ABI/API compatibility issues Trying to break out the build support itself might be useful. What do you mean by breakout exactly ? I have documented the already implemented support: http://docs.scipy.org/doc/numpy/reference/distutils.html#building-installable-c-libraries I think this needs some thought. This would essentially be a c library of iterator code. C++ is probably an easier language for such things as it handles the classes and inlining automatically. Which is to say if I had to deal with a lot of iterators I might choose a different language for implementation. C++ is not an option for numpy (and if I had to chose another language compared to C, I would rather take D, or one language which outputs C in the spirit of vala :) ). I think handling iterators in C is OK: sure, it is a bit messy, because of the lack of namespace, template and operator overloading, but the increased portability and
Re: [Numpy-discussion] Is this a bug in numpy.distutils ?
Dave wrote: David Cournapeau david at ar.media.kyoto-u.ac.jp writes: Matthew Brett wrote: Hi, We are using numpy.distutils, and have run into this odd behavior in windows: Short answer: I am afraid it cannot work as you want. Basically, when you pass an option to build_ext, it does not affect other distutils commands, which are run before build_ext, and need the compiler (config in this case I think). So you need to pass the -c option to every command affected by the compiler (build_ext, build_clib and config IIRC). cheers, David I'm having the same problems! Running windows XP, Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]. In my distutils.cfg I've got: [build] compiler=mingw32 [config] compiler = mingw32 Yes, config files are an alternative I did not mention. I never use them because I prefer controlling the build on a per package basis, and the interaction between command line and config files is not always clear. python setup.py build build_ext --compiler=mingw32 appeared to work (barring a warning: numpy\core\setup_common.py:81: MismatchCAPIWarning) The warning is harmless: it is just a reminder that before releasing numpy 1.4.0, we will need to raise the C API version (to avoid problems we had in the past with mismatched numpy version). There is no point updating it during dev time I think. but then how do I create a .exe installer afterwards? python setup.py bdist_wininst fails with the same error message as before and python setup.py bdist_wininst --compiler=mingw32 fails with the message: error: option --compiler not recognized You need to do as follows, if you want to control from the command line: python setup.py build -c mingw32 bdist_wininst That's how I build the official binaries . cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is this a bug in numpy.distutils ?
Dave wrote: David Cournapeau david at ar.media.kyoto-u.ac.jp writes: You need to do as follows, if you want to control from the command line: python setup.py build -c mingw32 bdist_wininst That's how I build the official binaries . cheers, David Running the command: C:\dev\src\numpypython setup.py build -c mingw32 bdist_wininst build.txt still gives me the error: error: Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing -c mingw32 to setup.py. I tried without a distutils.cfg file and deleted the build directory both times. In case it helps the bulid log should be available from http://pastebin.com/m607992ba Am I doing something wrong? No, I think you and Matthew actually found a bug in recent changes I have done in distutils. I will fix it right away, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is this a bug in numpy.distutils ?
On Tue, Aug 4, 2009 at 5:28 PM, David Cournapeauda...@ar.media.kyoto-u.ac.jp wrote: No, I think you and Matthew actually found a bug in recent changes I have done in distutils. I will fix it right away, Ok, not right away, but could you check that r7280 fixed it for you ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is this a bug in numpy.distutils ?
Dave wrote: Dave dave.hirschfeld at gmail.com writes: Work's for me. -Dave Except now when trying to compile the latest scipy I get the following error: Was numpy installed from a bdist_wininst installer, or did you use the install method directly ? David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is this a bug in numpy.distutils ?
Dave wrote: David Cournapeau david at ar.media.kyoto-u.ac.jp writes: Dave wrote: Dave dave.hirschfeld at gmail.com writes: Work's for me. -Dave Except now when trying to compile the latest scipy I get the following error: Was numpy installed from a bdist_wininst installer, or did you use the install method directly ? David Numpy was installed with the bdist_wininst installer. In case it's relevant the installer seemed to create 2 egg-info files: numpy-1.4.0.dev7277-py2.5.egg-info numpy-1.4.0.dev7280-py2.5.egg-info Deleting the numpy directory and the egg-info files and re-installing from the bdist_wininst installer gave the same result (with the above 2 egg-info files) Installing numpy with python setup.py install seemed to work (at least the npymath.ini file was now in the numpy\core\lib\npy-pkg-config folder) I think I understand the problem. Unfortunately, that's looks tricky to solve... I hate distutils. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] strange sin/cos performance
On Wed, Aug 5, 2009 at 12:14 AM, Andrew Friedleyafrie...@indiana.edu wrote: Do you know where this conversion is, in the code? The impression I got from my quick look at the code was that a wrapper sinf was defined that just calls sin. I guess the typecast to float in there will do the conversion Exact. Given your CPU, compared to my macbook, it looks like the float32 is the problem (i.e. the float64 is not particularly fast). I really can't see what could cause such a slowdown: the range over which you evaluate sin should not cause denormal numbers - just to be sure, could you try the same benchmark but using a simple array of constant values (say numpy.ones(1000)) ? Also, you may want to check what happens if you force raising errors in case of FPU exceptions (numpy.seterr(raise=all)). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is this a bug in numpy.distutils ?
On Tue, Aug 4, 2009 at 8:13 PM, David Cournapeauda...@ar.media.kyoto-u.ac.jp wrote: I think I understand the problem. Unfortunately, that's looks tricky to solve... I hate distutils. Ok - should be fixed in r7281. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value
Steven Coutts wrote: Sorry ignore this, I cleanded out numpy properly, re-installed 1.3.0 and the tests are all running now. Do you mean that if you build with debug information, everything else being equal, you cannot reproduce the crashes ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value
On Mon, Aug 3, 2009 at 6:32 PM, Steven Couttsste...@couttsnet.com wrote: David Cournapeau wrote: Do you mean that if you build with debug information, everything else being equal, you cannot reproduce the crashes ? cheers, David That does appear to be the case, SciPy 1.7.0 is now also running fine. It is just getting weirder - the fact that numpy worked with bare BLAS/LAPACK and crashed with atlas lead me to think that it was an atlas problem. But now, this smells more like a compiler problem. I would first really check that the only difference between crash vs. no crash is debug vs non debug (both with ATLAS), to avoid chasing wrong hints. Practically, I would advised you to clone the numpy sources (one debug, one non debug), and build from scratch with a script to do things in a repeatable manner. Then, if indeed you have crash only with non debug build, I would recommend to install my project numscons, to be able to play with flags, and first try building with the exact same flags as a normal build, but adding the -g flag. For example: CFLAGS=-O2 -fno-strict-aliasing -g python setupscons.py install --prefix=blabla Hopefully, you will be able to reproduce the crash and get a backtrace, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] strange sin/cos performance
On Mon, Aug 3, 2009 at 10:32 PM, Andrew Friedleyafrie...@indiana.edu wrote:
While working on GSoC stuff I came across this weird performance behavior
for sine and cosine -- using float32 is way slower than float64. On a 2ghz
opteron:
sin float32 1.12447786331
sin float64 0.133481025696
cos float32 1.14155912399
cos float64 0.131420135498
Which OS are you on ? FWIW, on max os x, with recent svn checkout, I
get expected results (float32 ~ twice faster).
The times are in seconds, and are best of three runs of ten iterations of
numpy.{sin,cos} over a 1000-element array (script attached). I've produced
similar results on a PS3 system also. The opteron is running Python 2.6.1
and NumPy 1.3.0, while the PS3 has Python 2.5.1 and NumPy 1.1.1.
I haven't jumped into the code yet, but does anyone know why sin/cos are
~8.5x slower for 32-bit floats compared to 64-bit doubles?
My guess would be that you are on a platform where there is no sinf,
and our sinf replacement is bad for some reason.
Side question: I see people in emails writing things like 'timeit foo(x)'
and having it run some sort of standard benchmark, how exactly do I do that?
Is that some environment other than a normal Python?
Yes, that's in ipython.
cheers,
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] strange sin/cos performance
On Mon, Aug 3, 2009 at 11:08 PM, Andrew Friedleyafrie...@indiana.edu wrote:
Thanks for the quick responses.
David Cournapeau wrote:
On Mon, Aug 3, 2009 at 10:32 PM, Andrew Friedleyafrie...@indiana.edu wrote:
While working on GSoC stuff I came across this weird performance behavior
for sine and cosine -- using float32 is way slower than float64. On a 2ghz
opteron:
sin float32 1.12447786331
sin float64 0.133481025696
cos float32 1.14155912399
cos float64 0.131420135498
Which OS are you on ? FWIW, on max os x, with recent svn checkout, I
get expected results (float32 ~ twice faster).
The numbers above are on linux, RHEL 5.2. The PS3 is running Fedora 9 I
think.
I know next to nothing about the PS3 hardware, but I know that it is
quite different compared to conventional x86 CPU. Does it even have
both 4 and 8 bytes native float ?
Much more reasonable, but still not what I'd expect or what you seem to
expect.
On a x86 system with sinf available in the math lib, I would expect
the float32 to be faster than float64. Other than that, the exact
ratio depends on too many factors (sse vs x87 usage, cache size,
compiler, math library performances). One order magnitude slower seems
very strange in any case.
The times are in seconds, and are best of three runs of ten iterations of
numpy.{sin,cos} over a 1000-element array (script attached). I've produced
similar results on a PS3 system also. The opteron is running Python 2.6.1
and NumPy 1.3.0, while the PS3 has Python 2.5.1 and NumPy 1.1.1.
I haven't jumped into the code yet, but does anyone know why sin/cos are
~8.5x slower for 32-bit floats compared to 64-bit doubles?
My guess would be that you are on a platform where there is no sinf,
and our sinf replacement is bad for some reason.
I think linux has sinf, is there a quick/easy way to check if numpy is
using it?
You can look at the config.h in numpy/core/include/numpy, and see if
there is a HAVE_SINF defined (for numpy = 1.2.0 at least).
cheers,
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Funded work on Numpy: proposed improvements and request for feedback
Hi All, I (David Cournapeau) and the people at Berkeley (Jarrod Millman, Fernando Perez, Matthew Brett) have been in discussion so that I could do some funded work on NumPy/SciPy. Although they are obviously interested in improvements that help their own projects, they are willing to make sure the work will impact numpy/scipy as a whole. As such we would like to get some feedback about the proposal. There are several areas we discussed about, but the main 'vision' is to make more of the C code in numpy reusable to 3rd parties, in particular purely computational (fft, linear algebra, etc...) code. A first draft of the proposal is pasted below. Comments, request for details, objections are welcomed, Thank you for your attention, The Berkeley team, Gael Varoquaux and David Cournapeau == Proposal for improvements to numpy == NumPy is a solid foundation for efficient numerical computation with the python programming language. It consists in a set of extensions to add a powerful multi-dimensional array object. SciPy is built upon NumPy to add more high level functionalities such as numerical integration, linear algebra, statistical functions, etc\.\.\. Although the numpy codebase is mature, and can be reused both at the C and Python levels, there are some limitations in the numpy codebase which prevent some functionalities from being reused by third parties. This means that users of numpy either need to reimplement the functionalities, or to use workarounds. The main goal of this proposal is to improve numpy to circumvent those limitations in general manner. Reusable C libraries A lot of NumPy and SciPy code is in a compiled language (mostly C and Fortran). For computational code, it is generally advisable to split it into a purely computational code and a wrapping part, marshalling back and forth python objects/structures into basic C types. For example, when computing the exponential of the items in an array, most of NumPy's job is to extract the data from the array into one of the basic C type (int, double, etc...), call the C function exp, and marshall the data back into python objects. Making the marshalling and purely computational code separate has several advantages: 1. The code is easier to follow 2. The purely computational code could be reused by third parties. For example, even for simple C math functions, there is a vast difference in platform/toolchains support. NumPy makes sure that functions to handle special float values (NaN, Inf, etc...) work on every supported platform, in a consistent manner. Making those functions available to third parties would enable developers to reuse this portable functions, and stay consistent even on platforms they don't care about. 3. Working on optimizing the computational code is easier. 4. It would enable easier replacement of the purely computational code at runtime. For example, one could imagine loading SSE-enabled code if the CPU supports SSE extensions. 5. It would also helps for py3k porting, as only the marshalling code would need to change To make purely computational code available to third parties, two things are needed: 1. the code itself needs to make the split explicit. 2. there needs to be support so that reusing those functionalities is as painless as possible, from a build point of view (Note: this is almost done in the upcoming numpy 1.4.0 as long as static linking is OK). Splitting the code -- The amount of work is directly proportional to the amount of functions to be made available. The most obvious candidates are: 1. C99 math functions: a lot of this has already been done. In particular math constants, and special values support is already implemented. Almost every real function in numpy has a portable npy\_ implementation in C. 2. C99-like complex support: this naturally extends the previous series. The main difficult is to support platforms without C99 complex support, and the corresponding C99 complex functions. 3. FFT code: there is no support to reuse FFT at the C level at the moment. 4. Random: there is no support either 5. Linalg: idem. Build support - Once the code itself is split, there needs some support so that the code can be reused by third-parties. The following issues need to be solved: 1. Compile the computational code into shared or static library 2. Once built, making the libraries available to third parties (distutils issues). Ideally, it should work in installed, in-place builds, etc\.\.\. situations. 3. Versioning, ABI/API compatibility issues Iterators = When dealing with multi-dimensional arrays, the best abstraction to deal with indexing in a dimension-independent way is to use iterator. NumPy already has some iterators to walk into every item of an array, or every item but one axis. More general iterators are useful
Re: [Numpy-discussion] Is this a bug in numpy.distutils ?
Matthew Brett wrote: Hi, We are using numpy.distutils, and have run into this odd behavior in windows: I have XP, Mingw, latest numpy SVN, python.org python 2.6. All the commands below I am running from within the 'numpy' root directory (where 'numpy' is a subdirectory). If I run python setup.py build I get the following expected error: ''' No module named msccompiler in numpy.distutils; trying from distutils error: Unable to find vccarsall.bat ''' because, I don't have MSVC. If I run: python setup.py build -c mingw32 - that works. But. running python setup.py build_ext -c mingw32 generates the same error as above. Similarly: python setup.py build_ext -c completely_unknown Ignores the attempt to set the 'completely_unknown' compiler, whereas python setup.py build -c completely_unknown raises a sensible error. I conclude that the numpy.distutils build_ext command is ignoring at least the compiler options. Is that correct? Short answer: I am afraid it cannot work as you want. Basically, when you pass an option to build_ext, it does not affect other distutils commands, which are run before build_ext, and need the compiler (config in this case I think). So you need to pass the -c option to every command affected by the compiler (build_ext, build_clib and config IIRC). Long answer: The reason is linked to the single most annoying feature of distutils: distutils fundamentally works by running some commands, one after the other. Commands have subcommands. For Numpy, as far as compiled code is concerned, it goes like this: config - build - build_clib - build_ext (the build command calls all the subcommands build_* and config). Now, each command options set is independent on the other (build_ext vs. config in this case), but if you pass an option to a command it affects all its subcommands I believe. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value
Steven Coutts wrote: Steven Coutts stevec at couttsnet.com writes: I am getting this error when trying to run a script using scipy. Python 2.5 atlas-3.9.0 lapack-3.2 numpy-1.3.0 scipy-0.7.1 Anyone any ideas how I can fix this? Lapack 3.2 is problematic, so I would try to downgrade to 3.1.1 first. Which OS are you on ? The exact core you are running, as well as the output of the test suite (numpy.test()) would be helpful, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value
Steven Coutts wrote: David Cournapeau wrote: Lapack 3.2 is problematic, so I would try to downgrade to 3.1.1 first. Which OS are you on ? The exact core you are running, as well as the output of the test suite (numpy.test()) would be helpful, cheers, David Ok, I have downgraded to lapack 3.1.1 and re-compiled and re-installed everything. Numpy test seg faults :( http://pastebin.com/d56fd6e44 Not sure it can be considered as progress :) Could you run the test suite with verbosity set to ten (numpy.test(verbose=10)) ? Also, which fortran compiler are you using ? g77 or gfortran ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value
Steven Coutts wrote: Ok, downgraded numpy to 1.2.1 and all the tests pass now! That's really strange - Linux is the most tested configuration, numpy 1.3.0 should run without problems. There is something unusual with your configuration that I am missing. Could you build numpy 1.3.0 from scratch and paste the output (rm -rf build python setup.py build build.log) ? If you are willing to do it, I would also be interested whether numpy works ok if linked against BLAS/LAPACK instead of atlas (i.e. build numpy, again from scratch, with ATLAS=None python setup.py build, and then run the test suite). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value
Steven Coutts wrote: Steven Coutts stevec at couttsnet.com writes: Ok, downgraded numpy to 1.2.1 and all the tests pass now! Regards But now scipy seg faults straight away http://pastebin.com/de13dd62 Downgraded scipy to 0.7.0 and still the same seg fault :( Make sure your scipy is built against numpy 1.2.1. You cannot build scipy against numpy 1.3.0, and then use scipy with numpy 1.2.1 (but you can build against numpy 1.2.1 and use scipy with numpy 1.3.0). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows64bit ?
Dinesh B Vadhia wrote: A suggestion: How about releasing Numpy for the AMD64 version first (without Scipy) and then follow up with a later release with Scipy support? This would satisfy Numpy-only users which can't be a bad thing rather than having a version that is not usable (I believe) by either Numpy or Scipy users. Because scipy needs code I would pull out from numpy. You can build numpy for amd64 by yourself at the moment, BTW. I am afraid that distributing one numpy binary now would only makes things more confusing when scipy for amd64 will be out, as it will not work with this numpy, but only with a later one. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value
Steven Coutts wrote: David Cournapeau david at ar.media.kyoto-u.ac.jp writes: If you are willing to do it, I would also be interested whether numpy works ok if linked against BLAS/LAPACK instead of atlas (i.e. build numpy, again from scratch, with ATLAS=None python setup.py build, and then run the test suite). Yes that appears to work fine, all tests run. So that's a problem with ATLAS. Maybe a gcc bug ? Another user contacted me privately for my rpm repository, and got exactly the same problem with the rpms, on CENTOS 5.3 as well. I will try to look at it on a centos VM if I have time this WE, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to configure f2py on windows XP?
Chris Colbert wrote: unless you have a visual studio 2003 compiler, you may have to use python 2.6. Mingw works, as well. This way, you don't have to care about which VS version to use for which python interpreter on which platform, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory layout of record arrays
On Fri, Jul 31, 2009 at 12:53 AM, Nicolas Rougiernicolas.roug...@loria.fr wrote: Hello, I've been using record arrays to create arrays with different types and since I'm doing a lot of computation on each of the different fields, the default memory layout does not serve my computations. Ideally, I would like to have record arrays where each field is a contiguous block of memory. I don't think you can do it with record arrays: one of the fundamental design choice of numpy array layout is that the data pointer points to one block of N items, where each item is described by the dtype. To have contiguous layout for each member of the structured dtype, you need two arrays with the corresponding dtype. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows 64bit ?
Hi Klaus, Klaus Noekel wrote: Dear folks, just over a month ago there was a thread about plans for numpy, and IIRC somebody had volunteered to try and put together a working AMD64 version with an installer. Since then I have not heard about the issue again - but I may have missed part of the discussion. Have the plans firmed up by now? Is there a tentative date for a beta or RC? In particular: how much hope is there for a reasonably usable AMD64 numpy under Windows? There were some discussion about pushing 1.4.0 'early', but instead, I think we let it slipped - one consequence is that there will be enough time for 1.4.0 to be released with proper AMD64 support on windows. The real issue is not numpy per-se, but making scipy work on top of numpy in 64 bits mode. It is hard to give an exact date as to when those issues will be fixed, but it is being worked on. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New test failures
Hi Nils, Nils Wagner wrote: numpy.__version__ '1.4.0.dev7270' Python 2.5.1 on 64-bit box AssertionError: Items are not equal: ACTUAL: [[-0.28785625-0.21230127j 2.13664055+3.62986112j] [ 0.20296739+0.16528448j 4.73750353+6.42351294j]] DESIRED: [[-0.28785625-0.21230127j 2.13664031+3.62986112j] [ 0.20296741+0.16528448j 4.73750353+6.42351341j]] I am working on it - I think some of those errors are buggy unit tests, though - for a single precision complex number, actual and desired only differ in on ULPS in the above case, for example. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Commit privileges for Darren Dale
On Tue, Jul 28, 2009 at 9:47 PM, Darren Daledsdal...@gmail.com wrote: On Mon, Jul 27, 2009 at 12:06 AM, Charles R Harrischarlesr.har...@gmail.com wrote: Hi All, In case it got buried in the thread, Darren is asking for commit privileges. I think it's a good idea. Thank you for saying so. What can I do to help move this to the point where I can actually start committing? I would like to get my array_prepare patch into svn soon so I can bundle a new beta of Quantities in time for scipy-09. I'm going to be on vacation July 31-August 9, should I wait until I get back before checking it in (assuming I'm granted commit rights)? Why not putting your code under a svn branch or a bzr/git/whatever import of the trunk, take care of it, and then committing it after you come back ? I am unfortunately not in position to comment on your code, I am not familiar on this part of numpy, but I would like someone else to take 'responsibility' that your code is OK if possible. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] installation issue with a non-standard location of a python build
On Wed, Jul 29, 2009 at 12:38 AM, Daniel Wheelerdaniel.wheel...@gmail.com wrote: Hi, I'm having an installation issue building numpy 1.3.0 using python 2.6.2 which has been built in my local area in a non-standard place and was configured with --prefix=/users/wd15/usr/i686/4.0. When I run numpy's config (python setup.py config) everything seems to be fine. However, when I run python setup.py build, it gives the following error: Can you post the full output of python setup.py build ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] installation issue with a non-standard location of a python build
On Wed, Jul 29, 2009 at 2:38 AM, Daniel Wheelerdaniel.wheel...@gmail.com wrote: -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc: error trying to exec 'cc1': execvp: No such file or directory gcc: error trying to exec 'cc1': execvp: No such file or directory That's your problem right there: your compiler is broken. What happens if you call gcc by yourself on a trivial C program ? David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Why does assert_array_almost_equal sometimes raise ValueError instead of AssertionError ?
Hi, In some cases, some of the testing functions assert_array_* raise a ValueError instead of AssertionError: np.testing.assert_array_almost_equal(np.array([1, 2, np.nan]), np.array([1, 2, 3])) # raises ValueError np.testing.assert_array_almost_equal(np.array([1, 2, np.inf]), np.array([1, 2, 3])) # raises AssertionError This seems at least inconsistent - is there a rationale or should this be considered as a bug ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB
Kim Hansen wrote: From my (admittedly ignorant) point of view it seems like an implementation detail for me, that there is a problem with some intermediate memory address space. Yes, it is an implementation detail, but as is 32 vs 64 bits :) My typical use case would be to access and process the large filemapped, readonly recarray in chunks of up to 1,000,000 records 100 bytes each, or for instance pick every 1000th element of a specific field. That is data structures, which I can easily have in RAM while working at it. I think it would be cool to have an alternative (possible readonly) memmap implementation (filearray?), which is not just a wrapper around mmap.mmap (with its 32 bit address space limitation), but which (simply?) operates directly on the files with seek and read. I think that could be very usefull (well for me at least, that is). In my specific case, I will probably now proceed and make some poor mans wrapping convenience methods implementing just the specific featuires I need as I do not have the insight to subclass an ndarray myself and override the needed methods. In that manner I can go to 2GB still with low memory usage, but it will not be pretty. I think it would be quite complicated. One fundamental limitation of numpy is that it views a contiguous chunk of memory. You can't have one numpy array which is the union of two memory blocks with a hole in between, so if you slice every 1000 items, the underlying memory of the array still needs to 'view' the whole thing. I think it is not possible to support what you want with one numpy array. I think the simple solution really is to go 64 bits, that's exactly the kind of things it is used for. If your machine is relatively recent, it supports 64 bits addressing. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB
Kim Hansen wrote: The machine is new and shiny with loads of processing power and many TB of HDD storage. I am however bound to 32 bits Win XP OS as there are some other costum made third-party and very expensive applications running on that machine (which generate the large files I analyze), which can only run on 32 bits, oh well Windows 64 bits can run 32 bits applications - very few applications are 64 bits to this day (For example, even the most recent visual studio (2008) do not run in 64 bits AFAIK). But scipy does not work on windows 64 bits ATM - although numpy does without problem if you build it by yourself. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Installed C libraries and using core math library
Hi, I have recently integrated my work on numpy.distutils to build so called installable C libraries, that is pure C libraries which can be installed and reused by 3rd parties. The general documentation is in the distutils section: http://docs.scipy.org/doc/numpy/reference/distutils.html#building-installable-c-libraries Example to reuse the math library is given in the core math library section in the C api chapter: http://docs.scipy.org/doc/numpy/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures on FreeBSD buildbot
On Tue, Jul 28, 2009 at 8:44 AM, Charles R Harrischarlesr.har...@gmail.com wrote: ERROR: test_nan_items (test_utils.TestApproxEqual) -- Traceback (most recent call last): File /tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py, line 248, in test_nan_items self._assert_func(anan, anan) File ../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py, line 433, in assert_approx_equal ValueError: math domain error -- Ran 2177 tests in 30.179s I expect there is a problem with the FreeBSD math libraries. Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) Reading the code for assert_approx_equal, using log/pow functions on nan should produce this error. I will work on a fix, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures on FreeBSD buildbot
Charles R Harris wrote: On Mon, Jul 27, 2009 at 9:37 PM, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: On Tue, Jul 28, 2009 at 8:44 AM, Charles R Harrischarlesr.har...@gmail.com mailto:charlesr.har...@gmail.com wrote: ERROR: test_nan_items (test_utils.TestApproxEqual) -- Traceback (most recent call last): File /tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py, line 248, in test_nan_items self._assert_func(anan, anan) File ../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py, line 433, in assert_approx_equal ValueError: math domain error -- Ran 2177 tests in 30.179s I expect there is a problem with the FreeBSD math libraries. Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) Reading the code for assert_approx_equal, using log/pow functions nan should produce this error. I will work on a fix, Yeah, the function is kind of a mess. Yes - I am afraid I have added to the mess with my nan/inf handling :( I tried reimplementing from a clean state, but quite a few unit tests depend on some 'obscure' features. For example, do we really want assert_array_equal(np.array([1, 1]), 1) to be true ? I understand it is handy, but it can hide bugs, and it makes writing the function more complicated (as you can't check that both arrays have the same shape, and covering every corner case with handling scalar/scalar arrays/arrays is difficult). I am also concerned with speed issues (they are likely to be a bottleneck in many unit tests). I think most of the math can be avoided. Do you know of a way to avoid log/pow for approx equal ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures on FreeBSD buildbot
Charles R Harris wrote: I'd just look at the difference and see if it exceeded some fraction of the expected value. There is the problem of zero, which could be handled in the usual way as diff abserr + relerr. I think abserr would need to be a new keyword with a default value. Since the relerr is specified by digits you probably need 1/10**digits, but there should be no problem with that. Really, only a multiplication should be needed to determine the precision to x digits. Ok, I will have a shot at this. Nan and +/- inf need to be special cased but I haven't thought about that. They are a constant source of frustration. It is like it defies my brain, and I end up doing very stupid things after working for too long with nan and inf, especially when complex numbers are involved :) I agree that the broadcasting is both a convenience and a trap here, but its probably a bit late to change that. I thought about having an additional assert_exactly_equal: both arguments will need to have the same shape and type (dtype for arrays), negative zero and positive zero will be different as well. I also have another function which may be useful in one of my scipy branch for scipy.special: having the max error in term of ULPS. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB
Kim Hansen wrote: 2009/7/24 Citi, Luca lc...@essex.ac.uk: Hello! I have access to both a 32bit and a 64bit linux machine. I had to change your code (appended) because I got an error about not being able to create a mmap larger than the file. Here are the results... On the 32bit machine: lc...@xps2:~$ python /tmp/ppp.py Created 1 writeable mmaps containing 100 MB Created 2 writeable mmaps containing 200 MB Created 3 writeable mmaps containing 300 MB Created 4 writeable mmaps containing 400 MB Created 5 writeable mmaps containing 500 MB [..] Created 24 writeable mmaps containing 2400 MB Created 25 writeable mmaps containing 2500 MB Created 26 writeable mmaps containing 2600 MB Created 27 writeable mmaps containing 2700 MB Created 28 writeable mmaps containing 2800 MB Created 29 writeable mmaps containing 2900 MB Created 30 writeable mmaps containing 3000 MB Removing mmaps... Done... Traceback (most recent call last): File /tmp/ppp.py, line 19, in module mm = mmap.mmap(f.fileno(), 0) mmap.error: [Errno 12] Cannot allocate memory On the 64bit machine I can create 510 mmaps both with bytes_per_mmap at 100MiB and 1GiB: Created 1 writeable mmaps containing 1000 MB Created 2 writeable mmaps containing 2000 MB Created 3 writeable mmaps containing 3000 MB Created 4 writeable mmaps containing 4000 MB Created 5 writeable mmaps containing 5000 MB Created 6 writeable mmaps containing 6000 MB [..] Created 501 writeable mmaps containing 501000 MB Created 502 writeable mmaps containing 502000 MB Created 503 writeable mmaps containing 503000 MB Created 504 writeable mmaps containing 504000 MB Created 505 writeable mmaps containing 505000 MB Created 506 writeable mmaps containing 506000 MB Created 507 writeable mmaps containing 507000 MB Created 508 writeable mmaps containing 508000 MB Created 509 writeable mmaps containing 509000 MB Created 510 writeable mmaps containing 51 MB Removing mmaps... Done... Traceback (most recent call last): File /tmp/ppp.py, line 19, in module mm = mmap.mmap(f.fileno(), 0) mmap.error: [Errno 24] Too many open files I do not even have 510GiB free in the disk. But I think that is because the ext3 filesystem allows sparse files. I think this shows that the maximum mapped space cannot be more than the maximum address space which is 2**64 for 64bit machines, 2GiB for windows32 and 3GiB for linux32. Under WindowsXP, you can try to increase it from 2GiB to 3GiB using the /3GB switch in the boot.ini Best, Luca Hi Luca, thx for trying. Your test clearly shows that 32 bits imposes a severe limitation. I tried adding the /3GB switch to boot.ini as you suggested: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=Microsoft Windows XP Professional /noexecute=optin /fastdetect /3GB and rebooted the system. Unfortunately that did not change anything for me. I still hit a hard deck around 1.9 GB. Strange. The 3Gb thing only works for application specifically compiled for it: http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx I somewhat doubt python is built with this, but you could check this in python sources to be sure, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB
Kim Hansen wrote: I tried adding the /3GB switch to boot.ini as you suggested: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=Microsoft Windows XP Professional /noexecute=optin /fastdetect /3GB and rebooted the system. Unfortunately that did not change anything for me. I still hit a hard deck around 1.9 GB. Strange. The 3Gb thing only works for application specifically compiled for it: http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx I somewhat doubt python is built with this, but you could check this in python sources to be sure, cheers, David Ahh, that explains it. Thank you for that enlightening link. Anyway would it not be worth mentioning in the memmap documentation that there is this 32 bit limitation, or is it so straightforwardly obvious (it was not for me) that his is the case? Well, the questions has popped up a few times already, so I guess this is not so obvious :) 32 bits architecture fundamentally means that a pointer is 32 bits, so you can only address 2^32 different memory locations. The 2Gb instead of 4Gb is a consequence on how windows and linux kernels work. You can mmap a file which is bigger than 4Gb (as you can allocate more than 4Gb, at least in theory, on a 32 bits system), but you cannot 'see' more than 4Gb at the same time because the pointer is too small. Raymond Chen gives an example on windows: http://blogs.msdn.com/oldnewthing/archive/2004/08/10/211890.aspx I don't know if it is possible to do so in python, though. The reason it isn't obvious for me is because I can read and manipulate files 200 GB in Python with no problems (yes I process that large files), so I thought why should it not be capable of handling quite large memmaps as well... Handling large files is no problem on 32 bits: it is just a matter of API (and kernel/fs support). You move the file location using a 64 bits integer and so on. Handling more than 4 Gb of memory at the same time is much more difficult. To address more than 4Gb, you would need a segmented architecture in your memory handling (with a first address for a segment, and a second address for the location within one segment). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] sorting and nans, timings.
Charles R Harris wrote: Hi All, I changed the sort routines to sort nans to the end and got some timings. Sorting 10 random doubles 100 times yields: current nan version quicksort 1.17 sec1.29 sec mergesort 1.37 sec1.36 sec heapsort 1.83 sec2.12 sec Curiously, mergesort doesn't seem to suffer at all. This is using x != x for nan detection, using npy_isnan is notably slower with my compiler (gcc 4.3.0). That's because glibc isnan is slow :) I was surprised, but our own isnan replacement is much faster than the glibc one at least (like almost twice as fast on my P4 machine). Glibc isnan avoids branching, but it seems that it tries too hard. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] sorting and nans, timings.
David Cournapeau wrote: Charles R Harris wrote: Hi All, I changed the sort routines to sort nans to the end and got some timings. Sorting 10 random doubles 100 times yields: current nan version quicksort 1.17 sec1.29 sec mergesort 1.37 sec1.36 sec heapsort 1.83 sec2.12 sec Curiously, mergesort doesn't seem to suffer at all. This is using x != x for nan detection, using npy_isnan is notably slower with my compiler (gcc 4.3.0). That's because glibc isnan is slow :) I was surprised, but our own isnan replacement is much faster than the glibc one at least (like almost twice as fast on my P4 machine). Glibc isnan avoids branching, but it seems that it tries too hard. IMO, a 10-15 % increase in time for better nan handling in sort definitely worths it. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pdf for multivariate normal function?
per freem wrote: hi all, i'm trying to find the function for the pdf of a multivariate normal pdf. i know that the function multivariate_normal can be used to sample from the multivariate normal distribution, but i just want to get the pdf for a given vector of means and a covariance matrix. is there a function to do this? AFAIK, there is no such function in scipy. There is one in the em subpackage of the learn scikits. The file which implements it is self-contained, so you could use this in the meantime: http://projects.scipy.org/scikits/browser/trunk/learn/scikits/learn/machine/em/densities.py I wrote this code some time ago, so it is not optimal in various ways, but it should get the job done. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/core/code_generators/../src/multiarray/iterators.c:1778:could not find function name
Nils Wagner wrote: Hi all, I cannot build numpy from svn. Yes, I don't know why I did not caught this error on my machine. In any case, it is fixed in r7175. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding new functions to Numpy
Hi Pauli, On Sat, Jul 4, 2009 at 9:59 PM, Pauli Virtanenpav...@iki.fi wrote: Hi, When you add new functions to Numpy, please include .. versionadded:: 1.4.0 What is the best way to do this in the reference guide directly as well (for C API). For example, I added the function npy_copysign for 1.4.0, documented in the c-api.coremath.rst, but I don't know where to put the versionadded. More generally, I would like to follow our docstring format when documenting C functions, but how can I put the different sections in there (Parameters, Notes, etc...) cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy for 2.6 on mac os x
David Cournapeau wrote: On Sat, Jul 11, 2009 at 12:19 AM, Tommy Gravtg...@mac.com wrote: The current dmg on the numpy download pages is buildt against 2.5. Is there any plans to make one for 2.6 or do I have to compile from the source? A 2.6 dmg is now available on sourceforge. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy for 2.6 on mac os x
On Sat, Jul 11, 2009 at 12:19 AM, Tommy Gravtg...@mac.com wrote: The current dmg on the numpy download pages is buildt against 2.5. Is there any plans to make one for 2.6 or do I have to compile from the source? There are plans :) I am building the 0.7.1 binaries right now, and mac os x binaries will be available for both 2.6 and 2.5. I will add the corresponding 2.6 numpy binary at the same time. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)
Matthieu Brucher wrote: Unfortunately, this is not possible. We've been playing with blocking loops for a long time in finite difference schemes, and it is always compiler dependent You mean CPU dependent, right ? I can't see how a reasonable optimizing compiler could make a big difference on cache effects ? @ Pauli: if (optionally) knowing a few cache info would help you, I could implement it. It should not be too difficult for most cases we care about, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Question for David re npy_funcs
On Wed, Jul 8, 2009 at 5:37 AM, Charles R Harrischarlesr.har...@gmail.com wrote: David, Should any standard c functions used in loops.c.src be the npy_* version? I've been using fabs, but I'm wondering if that should be npy_fabs. Yes. Although fabs is available on any platform in theory, we should standardize on using npy_ everywhere (even if the function is available, we may use a different implementation for the npy_ one because the original is too buggy). It would be nice to enforce this, actually. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] RFC: add a install_clib numpy.distutils command to install pure C libraries
On Fri, Jun 12, 2009 at 7:46 PM, David Cournapeauda...@ar.media.kyoto-u.ac.jp wrote: Hi, I have finally spent some time so that we can install pure C libraries using numpy.distutils. With this, one could imagine having a C library for fft, special functions in numpy or scipy, so that the library could be reused in another package at the C level. If someone knowledgeable about numpy.distutils would like to review this branch, I would be grateful: I reworked the branch, making it simpler to add installable libraries, and ensuring it works for both conventional install and in-place builds. The design is better, I believe: - libraries are installed through the add_installed_library function, which is like add_library but with an install_dir argument - build_clib now takes an in-place argument (so that installable libraries can be reused from an in-place build) - retrieving build info has been totally redesigned. Instead of hardcoding it, I know use .ini file for each installable library, with a syntax similar to .pc files as used by pkg-config. - build information to reuse the installable libraries is through the get_info function, which returns a dict compatible with the extra_info argument of add_library/add_extension. The main change is thus the .ini file. The infrastructure could be used to replace the rather inflexible system_info at some point. It does not support every option of pkg-config, but it can handle dependencies. For example, npymath depends on the math library, so querying the libs option of npymath will automatically pull out the libs option of the mlib as well. They can also be queried from the command line: python numpy/distutils/npy_pkg_config npymath --cflags python numpy/distutils/npy_pkg_config npymath --libs So that non distutils-based build systems can retrieve the info easily (cmake, in particular). I think it would be nice to have this in 1.4.0, so that we can rely on npymath in scipy, and I need this for windows 64 bits support, cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)
On Thu, Jul 9, 2009 at 8:02 AM, Pauli Virtanenpav...@iki.fi wrote: I don't think we want to go the ATNumPy route, or even have tunable parameters chosen at build or compile time. Detecting things like cache size at compile time should not be too difficult, at least for common platforms. Even detecting it at runtime should be relatively simple in some particular cases (x86). BTW, one good baseline for those summation is to use dot: np.ones((8, 256)).sum(axis=0) vs np.dot(np.ones((1, 8)), np.ones((8, 256))) Assuming dot uses an optimized blas, this is generally one order of magnitude faster than sum. (Unless, of course, we want to bring a monster into the world -- think about cross-breeding distutils with the ATLAS build system :) Kill me now :) David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] The baffling behavior that just won't die
Pauli Virtanen wrote: Sun, 05 Jul 2009 20:27:02 -0700, d_l_goldsmith kirjoitti: [clip] c:\Python25\Lib\site-packages\numpypython Just don't run Python inside Numpy's package directory. This is not Numpy- specific: doing a thing like that just breaks relative imports. I noticed that many windows users make this mistake. I wonder why, and if there is something (in the docs, etc...) that we could improve to avoid this. I can't understand why anyone would got into site-packages ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] The baffling behavior that just won't die
David Goldsmith wrote: --- On Mon, 7/6/09, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: avoid this. I can't understand why anyone would got into site-packages ? Why, to look at the source, of course - I did it all the time in Mac Unix, too - must not have ever tried to run anything while in there I guess; maybe I knew this and forgot. Oh well, don't think I'll forget this time (just as I don't think I'll now forget what the qr factorization is all about). :-) Ah, I get it now - since few people build from sources on windows, looking there is the only way to look at the sources. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy complex types, packing and C99
On Thu, Jul 2, 2009 at 9:02 AM, David Cournapeaucourn...@gmail.com wrote: True, but we can deal with this once we have tests: we can force to use our own, fixed implementations on broken platforms. The glibc complex functions are indeed not great, I have noticed quite a few problems for special value handling (e.g. cabs(inf + I * NAN) is nan, but should be inf, etc). Ok, here we are. There are two branches. - the first one, complex_umath_tests, is a branch with thorough covering of special values. I use the C99 standard for reference: http://github.com/cournape/numpy/tree/complex_umath_tests The main file is http://github.com/cournape/numpy/blob/c29e793c220f48e317adafb8c765acd4db13cb0d/numpy/core/tests/test_umath_complex.py Around 10 tests fail on Linux ATM. - the second branch is the interesting one: http://github.com/cournape/numpy/tree/C99_complex_math It implements clog, cexp, creal, cimag, carg, cabs, ccos, csin and csqrt. The list is dictated by my needs for windows 64, but the list can grow arbitrarily, of course. Most the implementations are taken from msun (the math library of FreeBSD). Unfortunately, they don't implement that many functions compared to the glibc. If I merge the two branches and use the npymath complex functions in umath, there is no unit test failures anymore :) I think I will merge the complex_umath_tests branch soon (platform-specific failures on build bot will be interesting), unless someone sees a problem with it. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy complex types, packing and C99
On Fri, Jul 3, 2009 at 4:44 AM, Pauli Virtanenp...@iki.fi wrote: I think we tried this already (my c99-umath-funcs branch had TestC99 special case tests that were in Numpy trunk for a while). The outcome was that the current implementations of the complex functions don't have essentially any special-case behavior that's consistent across different platforms. And our buildbot coverage is not large enough, so users will report failing tests even if buildbots are OK... After repeating this cycle a couple of times, IIRC only some special cases of log survived :) I think the situation is better now, because we have the infrastructure to deal with this: we have portable nan, inf, negative zero and corresponding macros (I have just added copysign recently to the trunk, which was the last missing for every case AFAIK). Preliminary tests on both windows (with MS compilers) and linux indicate that many special cases are already dealt with correctly. I don't have tests for branch cuts yet, though, only signed 0, nan and inf handling. Of course, if you meant to merge the tests first to the new implementations and that to trunk, this sounds better. The idea was to merge all the tests, but set them as knownfailure, and enable them everytime I merge a new function into the trunk. This way, one can easily check the differences between successive revisions ? But that's not a crucial point. I can keep everything in one branch if you think it is better. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Numpy complex types, packing and C99
Hi, I would like to add an explicit configuration test to check that our complex type is compatible with the C99 complex type (when available). Is this ok ? As currently defined, complex c types (npy_cfloat, etc...) are not defined in a way such as they are binary compatible with the C99 complex type. Strictly speaking, packing the complex type is an ABI break, but we already make the assumption in ufunc, so we would have wrong results/crashes currently if it were not packed, so I believe the check is harmless by itself. The rationale is that I would like to add support for complex math in npy_math (cabs, cexp, etc...). As I would like to use the C99 complex type (when available), this requires that numpy complex type is ABI compatible with C99 type. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.memmap and memory usage
On Wed, Jul 1, 2009 at 6:14 PM, Pauli Virtanenp...@iki.fi wrote: Wed, 01 Jul 2009 10:17:51 +0200, Emmanuelle Gouillart kirjoitti: I'm using numpy.memmap to open big 3-D arrays of Xray tomography data. After I have created a new array using memmap, I modify the contrast of every Z-slice (along the first dimension) inside a for loop, for a better visualization of the data. Although I call memmap.flush after each modification of a Z-slice, the memory used by Ipython keeps increasing at every new iteration. At the end of the loop, the memory used by Ipython is of the order of magnitude of the size of the data file (1.8Go !). I would have expected that the maximum amount of memory used would corresponde to only one Z-slice of the 3D array. See the code snapshots below for more details. Is this an expected behaviour? How can I reduce the amount of memory used by Ipython and still process my data? How do you measure the memory used? Note that on Linux, top includes the size of OS caches for the memmap in the RSS size of a process. You can try to monitor free instead: A good tool on linux is exmap: http://ktown.kde.org/~seli/memory/analysis.html Unfortunately, it is static, but more accurate than free or top. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
