Re: [Numpy-discussion] Convert array type
Adam Mercer wrote: On 08/10/2007, Ryan May [EMAIL PROTECTED] wrote: Why not use numpy.fromstring? because that results in the array being filled with gibberish values = numpy.fromstring(wavearray, dtype=float, count=-1, sep='') Use sep=' '. As the docstring says, if sep is empty, then the string is interpreted as binary data. If it is not empty, then the string is interpreted as ASCII. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fortran 90 compiler problem
[EMAIL PROTECTED] wrote: Hello, I'm using the last svn version of numpy (. I tried the solution found on the mail list: python setup.py config_fc --fcompiler=gnu95 build the two interesting part (at least for me) in the building process are: customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config and later: customize G95FCompiler Found executable /usr/local/bin/g95 but I still have the same answer when I'm trying to do: f2py -c -m hello hello.f90 customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found This is a correct message and does not imply that no Fortran 90 compiler exists. The GnuFCompiler (gnu) only looks at g77, which is not a Fortran 90 compiler. That message only applies to the gnu compiler, not any other. and so: error: f90 not supported by GnuFCompiler needed for hello.f90 I can't compile my fortran module because of this. I'll attach the file obtain with python setup.py -v config_fc --fcompiler=gnu95 build And I have the same result for g95. thank you for numpy and f2py. Please give us the build log that actually failed and the command that you used to obtain it. The one you show, for numpy, succeeded. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fortran 90 compiler problem
[EMAIL PROTECTED] wrote: This is the log I have when I'm trying to use f2py with a fortran 90 file. The last line seems to tell me that f2py cannot see gfortran and g95 even if it detected them when I installed f2py. thanks, N. gfortran and g95 are installed and I can use them: gfortran gfortran: no input files gfortran -c hello.f90 ls hello* hello.f90 hello.o g95 g95: no input files g95 -c hello.f90 ls hello* hello.f90 hello.o and now f2py -c -m hello hello.f90 Tell f2py that you want it to use gfortran instead of the default. f2py --fcompiler=gnu95 -c -m hello hello.f90 -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fortran 90 compiler problem
Charles R Harris wrote: On 10/10/07, *Robert Kern* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Hello, I'm using the last svn version of numpy (. I tried the solution found on the mail list: python setup.py config_fc --fcompiler=gnu95 build the two interesting part (at least for me) in the building process are: customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config and later: customize G95FCompiler Found executable /usr/local/bin/g95 but I still have the same answer when I'm trying to do: f2py -c -m hello hello.f90 customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found This is a correct message and does not imply that no Fortran 90 compiler exists. The GnuFCompiler (gnu) only looks at g77, which is not a Fortran 90 compiler. That message only applies to the gnu compiler, not any other. I have always found these messages confusing. Perhaps they could be better worded to reflect the process, or, even better, silenced. All that really matters is success or failure and the reason, not every little dead end explored on the way. If they are needed for debugging, write the whole mess to a file in the build directory instead of the console. Go for it. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] appending extra items to arrays
Adam Mercer wrote: Hi In some code I have, I need to append some extra data to a given array. At the moment I construct the data in a list, append the extra information I need and then convert the final list to an array. Is there a way that I can append extra information to an existing array thereby negating the need for the list, as the array object doesn't seem to have an append() method? Appending to a list then converting the list to an array is the most straightforward way to do it. If the performance of this isn't a problem, I recommend leaving it alone. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] appending extra items to arrays
Anne Archibald wrote: On 11/10/2007, Robert Kern [EMAIL PROTECTED] wrote: Appending to a list then converting the list to an array is the most straightforward way to do it. If the performance of this isn't a problem, I recommend leaving it alone. Just a speculation: Python strings have a similar problem - they're immutable, and so are even more resistant to growth than numpy arrays. For those situations where you really really want to grow a srting, python provides StringIO, where you keep efficiently adding to the string, then finalize it to get the real string out. Would something analogous be interesting for arrays? The Python version of StringIO, at least, just keeps a list, too. The primary benefit of StringIO is that it exposes the file interface. For building up a string/array, using the list is just as convenient (and for arrays, probably more convenient given the variety of shapes one might want to build) and at least as readable. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote: David Cournapeau wrote: Pearu Peterson wrote: I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ? Not always. For example, in debian one has to install python-dev separately from python. You'd still need python-dev(el) for the headers if not distutils. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] .bytes
Yves Revaz wrote: Nadav Horesh wrote: array(1, dtype=float32).itemsize ok, it will work fine for my purpose. In numpy, is there any reason to supress the attribute .bytes from the type object itself ? Is it simply because the native python types (int, float, complex, etc.) do not have this attribute ? The problem is that the instances of the scalar types do have the itemsize attribute. The implementation of type objects is such that the type object will also have that attribute, but it will be a stub: In [15]: float64.itemsize Out[15]: attribute 'itemsize' of 'numpy.generic' objects A more straightforward way to get the itemsize is this: In [17]: dtype(float64).itemsize Out[17]: 8 -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A basic question on the dot function
Julien Hillairet wrote: Hello, First of all, I'm sorry if this question had already been asked. I've searched on the gmane archive and elsewhere on internet, but I didn't found the answer to my question. As expected, the dot product of 2 'classical' vectors works fine : In [50]: a0 = numpy.array([1,2,3]) In [51]: numpy.dot(a0,a0) Out[51]: 14 What I don't understand, is why the dot product of a (3,N) vector gives an error : In [52]: a1 = numpy.array([[1,2,3],[1,2,3],[1,2,3],[1,2,3]]) In [54]: numpy.dot(a1,a1) type 'exceptions.ValueError'Traceback (most recent call last) type 'exceptions.ValueError': objects are not aligned instead of what I've expected ; an array([14 14 14 14]). When given two 2-D arrays, dot() essentially does matrix multiplication. The last dimension of the first argument is matched with the next-to-last dimension of the second argument. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Sum of the product of three or more arrays
Geoffrey Zhu wrote: Hi All, Given three vectors of the same lengths, X, Y, and Z, I am looking for an efficient way to calculate the following: sum(x[i]*y[i]*z[i], for i=1..n ) (x*y*z).sum() -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] vectorizing loops
Gael Varoquaux wrote: On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote: Anybody know of any tricks for handling something like z[0]=1.0 for i in range(100): out[i]=func1(z[i]) z[i+1]=func2(out[i]) Something like: z[0] = 1. out = func1(z) z[1:] = func2(out[:-1]) No, that doesn't work. The way you have it, for each i0, z[i] = func2(func1(0)) What Matthew wants is this z[0] = 1.0 z[1] = func2(func1(1.0)) z[2] = func2(func1(func2(func1(1.0 ... I'm afraid that there is no fast, elegant way to do this. If you could turn func2(func1(x)) into a binary ufunc f(x, y) with an ignored y, then you could call f.accumulate(z). However, if that ufunc is not implemented in C, there's not much point. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] matrix problem: float to matrix power
Charles R Harris wrote: On 10/31/07, Robert Kern [EMAIL PROTECTED] wrote: Charles R Harris wrote: On 10/31/07, Alan G Isaac [EMAIL PROTECTED] wrote: 1.0**numpy.array([1,2,3]) array([ 1., 1., 1.]) 1.0**numpy.mat([1,2,3]) Traceback (most recent call last): File stdin, line 1, in module TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'matrix' Why the restriction for matrices? Looks like it just got overlooked. It works fine for 2D arrays. In [11]: 1.0**eye(3) Out[11]: array([[ 1., 1., 1.], [ 1., 1., 1.], [ 1., 1., 1.]]) For ndarrays, it does elementwise exponentiation, so it is neither here nor there with respect to matrices. What is the definition of a scalar raised to a matrix power? I don't ever recall seeing one. I suppose the usual: a**m = exp(ln(a)*m) but that isn't the broadcast version. Ah, yes. Of course. How silly of me. That raises the question of how to calculate the bloody thing, though. We have three implementations of expm() in scipy.linalg.matfuncs. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with numpy on Leopard
Matthieu Brucher wrote: Hi, The problem will arise for every package, not only numpy, so Apple fixing this is the best solution IMHO. It's unlikely they are going to. If they put that stuff there, it's because they are using it for something, not as an (in)convenience to you. I don't recommend using the Python.framework in /System for anything except for distributing lightweight .apps. In that case, you can control your sys.path very well. For regular work, use the binary from www.python.org. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Assessing the use of packages
Pierre GM wrote: All, It's evaluation time in my department (Bio. Ag. Engng, UGA), and I'd need to document the impact of my Python contributions on the scientific community at large, or more realistically on the numpy/scipy user community... * Is there a way to estimate how many people installed one particular package from the SVN ? No. We could only record who has checked them out from SVN. However, everyone who has checked out the scipy trunk will have gotten the packages you are concerned with. Whether or not they've built them or used them is another matter which we cannot determine. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1.04 numpy.test() hang
Geoffrey Zhu wrote: Good morning. I just installed the Windows binary of numpy 1.04. When I ran numpy.test() in IDLE (the Python shell that comes with Python), the program hang (or at least is running for half an hour). I am using Windows XP, duel core intel CPU. Does anyone know what is going on? No. Run numpy.test(verbosity=2) so it will print out each test name before running the test. Then we might have some idea about where the hang is coming from. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] weibull distribution has only one parameter?
D.Hendriks (Dennis) wrote: According to (for instance) http://en.wikipedia.org/wiki/Weibull_distribution the Weibull distribution has two parameters: lambda 0 is the scale parameter (real) and k 0 is the shape parameter (real). However, the numpy.random.weibull function has only a single 'a' parameter (except for the size parameter which indicates the size of the array to fill with values - this is NOT a parameter of the distribution itself). My question is how this 'a' parameter translates to the Weibull distribution as it 'normally' is and how to sample the distribution when I have the lambda and k parameters? lambda * numpy.random.weibull(k) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] weibull distribution has only one parameter?
D.Hendriks (Dennis) wrote: Alan G Isaac wrote: On Mon, 12 Nov 2007, D.Hendriks (Dennis) apparently wrote: All of this makes me doubt the correctness of the formula you proposed. It is always a good idea to hesitate before doubting Robert. URL:http://en.wikipedia.org/wiki/Weibull_distribution#Generating_Weibull-distributed_random_variates hth, Alan Isaac So, you are saying that it was indeed correct? That still leaves the question why I can't seem to confirm that in the figure I mentioned (red and green lines). Also, if you refer to X = lambda*(-ln(U))^(1/k) as 'proof' for the validity of the formula, I have to ask if Weibull(a,Size) does actually correspond to (-ln(U))^(1/a)? double rk_standard_exponential(rk_state *state) { /* We use -log(1-U) since U is [0, 1) */ return -log(1.0 - rk_double(state)); } double rk_weibull(rk_state *state, double a) { return pow(rk_standard_exponential(state), 1./a); } Like Ryan says, multiplying a random deviate by a number is different from multiplying the PDF by a number. Multiplying the random deviate by lambda is equivalent to transforming pdf(x) to pdf(x/lambda) not lambda*pdf(x). -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposing Ubiquity
dieter h. wrote: Hi all, Would it make sense for all functionality in Numpy/Scipy to have ubiquitous returns? In that I'm proposing that every func/method (where appropriate) have a flag in its arg list establishing a return for either [INDEX, BOOLEAN, VALUE=default]. this could be by a singular Enum flag, I believe. Obvioulsy, this proposal gives the option for a return of Index/Bool to be used for Indexing arrays, or the actual values. Since I mosty index existing arrays, this would be a huge plus for me. I'm afraid that I don't understand this proposal. Can you show some examples of what you would like to see? Preferably, can you write some working code that demonstrates this feature? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Should array iterate over a set?
Michael McNeil Forbes wrote: My expectation was that array would iterate over a set. This is incorrect: array(set([1,2,3])) array(set([1, 2, 3]), dtype=object) Is this the intended behaviour? A trivial work-around that does what I need is array(list(set([1,2,3]))) array([1, 2, 3]) but I was wondering if this was by design or just a forgotten corner. (Maybe a vestige of the tuple special case for record arrays?) We can recognize most sequences (i.e. for all i in range(len(x)), x[i] responds correctly), but we cannot easily deal with arbitrary iterables which are not sequences in the array() function. There are a lot of special cases and heuristics going on in array() as it is. Instead, we have a fromiter() function which will take an iterable and construct an array from it. It is limited to 1D arrays, but this is by far the most common use for constructing an array from an iterable. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC
Joshua Lippai wrote: I updated my GCC to a more recent version a day ago, since Apple's Xcode Tools only provide GCC 4.0 and the current release of GNU's GCC is 4.2. I successfully achieved this, but now I run into a problem when trying to build NumPy: gcc: unrecognized option '-no-cpp-precomp' cc1: error: unrecognized command line option -arch cc1: error: unrecognized command line option -arch cc1: error: unrecognized command line option -Wno-long-double gcc: unrecognized option '-no-cpp-precomp' cc1: error: unrecognized command line option -arch cc1: error: unrecognized command line option -arch cc1: error: unrecognized command line option -Wno-long-double Upon investigation into the matter, I found out that these options (no-cpp-precomp and Wno-long-double) are only valid in Apple's GCC and not the regular GNU release. Yet it seems NumPy automatically assumes Apple's GCC is being used when it realizes the target is OS X. Is there a way around this, or at least some way to specify Apple's GCC? NumPy is the only package I've tried building so far that has a problem with this. I'm surprised that you've built other Python extension modules because numpy does not add these flags; Python does. Python extensions should be built with the same compiler that Python itself was built with. If you are using the binary distribution from www.python.org, you should use Apple's gcc, not a different one. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC
Joshua Lippai wrote: Thanks for the reply. Well, I built my Python stuff, including NumPy previously, before I changed to the higher GCC version. Do you know if there's an option I can toggle that will specify Apple's GCC to be used? $ CC=/usr/bin/gcc python setup.py build -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] display numpy array as image
Zachary Pincus wrote: Hello all, I'm curious if people have experience with / preferences for how to display a numpy array onscreen as an image. Pyglet looks relatively easy -- you can feed an image buffer object with a string or a ctypes pointer. I presume getting a string from an array is plenty fast, but the ctypes pointer option is intriguing as it allows for dealing with simple strided arrays (the image objects allow for an arbitrary number of bytes between rows). Is it possible to get a ctypes pointer to the beginning of the array buffer from numpy without too much ugliness? In [16]: from numpy import * In [17]: a = arange(10) In [18]: dir(a.ctypes) Out[18]: ['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', '_arr', '_as_parameter_', '_ctypes', '_data', '_zerod', 'data', 'data_as', 'get_as_parameter', 'get_data', 'get_shape', 'get_strides', 'shape', 'shape_as', 'strides', 'strides_as'] In [22]: import ctypes In [24]: a.ctypes.data_as(ctypes.POINTER(ctypes.c_long)) Out[24]: ctypes.LP_c_long object at 0x1b353a0 In [25]: a.ctypes.get_shape() Out[25]: numpy.core._internal.c_long_Array_1 object at 0x1c096c0 In [26]: a.ctypes.get_strides() Out[26]: numpy.core._internal.c_long_Array_1 object at 0x1c09710 In [27]: a.ctypes.get_as_parameter() Out[27]: c_void_p(27442576) You might want to use the new ctypes-based OpenGL 3.0+ package. It has numpy support a bit more directly. You can use Pyglet for its windowing and all of the other surrounding infrastructure and use OpenGL directly for the drawing. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy binary for macosx tiger on ppc
Barry Wark wrote: Some remaining issues: - which SDK to build against. Leopard ships with a Python build against the 10.5 SDK. It would be much easier, at least initially, for us to produce binaries against the Leopard Python 2.5. I would prefer that we use the Python binary from www.python.org. That should work on 10.3.9+. - how to deal with library dependencies such as fftw2. We currently use MacPorts but I suppose it needs to be compiled statically or do we just require that users install MacPort's fftw2? Official binaries intended for distribution from scipy.org or scipy.sf.net should not be linked against FFTW or UMFPACK since they are GPLed. The real problem is the Fortran runtime. It is possible to link against the Fortran runtime libraries statically so as to avoid needing the user to also install a Fortran compiler just to run scipy. Use the gfortran compiler built for R's use (and do not use the one from hpc.sf.net): http://r.research.att.com/tools/ Basically, copy the libgfortran.a library to a separate location, say ~/staticlibs/. Then do the following: $ export LDFLAGS=-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first $ python setup.py config_fc --fcompiler=gnu95 --arch=-arch i386 -arch ppc build_ext -L ~/staticlibs/ build -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy binary for macosx tiger on ppc
Christopher Barker wrote: Robert Kern wrote: Official binaries intended for distribution from scipy.org or scipy.sf.net should not be linked against FFTW or UMFPACK since they are GPLed. Does that apply to binaries put up on pythonmac.org? It would be nice to have a complete version somewhere standard. I would argue yes, it does apply to pythonmac.org, too. scipy is a BSD-licensed package. Putting GPLed components into the binaries which are distributed at standard locations like our website, PyPI, or pythonmac.org confuses that picture. I would be mollified if the download link had an explanation attached, but unfortunately, pythonmac.org/packages does not have any space for that. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] random.hypergeometric bug
Chris wrote: There appears to be a bug in numpy's hypergeometric random number generator. Here is an example -- if I generate 1000 hg samples with 4 draws from a space with 30 successes and 10 failures: In [39]: x = hg(30, 10, 4, 1000) I should get a mean value of: In [40]: 4*30./40 Out[40]: 3.0 But the sample mean is way to small: In [41]: mean(x) Out[41]: 0.996 Fixed in r4527. My original source for the algorithm was incorrect, it seems. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
David Cournapeau wrote: Some of the most interesting things I can think of which work with scons: - you can control fortran and C flags from the command line: CFLAGS and FFLAGS won't override necessary flags, only optimization flags, so you can easily play with warning, optimization flags. For example: CFLAGS='-W -Wall -Wextra -DDEBUG' FFLAGS='-DDEBUG -W -Wall -Wextra' python setupscons build for debugging will work. No need to care about -fPIC and co, all this is handled automatically. Can I override the flags which are handled automatically without modifying numpy? I just spent much of last night trying to get Intel Fortran on OS X working, and I had to dive into numpy.distutils.fcompiler.intel a lot. This is mildly acceptable, if irritating, for a numpy developer, but not acceptable for even a sophisticated user. Even if we were to keep our knowledge-base of Fortran compiler flags immediately up-to-date with every release of every Fortran compiler we follow, people will still be stuck with older versions of numpy. numpy.distutils' behavior of using LDFLAGS, for example, to completely replace the flags instead of extending them mitigated this, somewhat. It allowed someone to work around the stale flags in numpy.distutils in order to get something built. This is a hack, and it causes confusion when this isn't the desired behavior, but it worked. But can we do something better with scons? One option which would work both with scons and the current numpy.distutils is to provide something like LDFLAGS_NO_REALLY which replaces the flags and let LDFLAGS just extend the flags. That doesn't help the ultimate problem, but it makes the workaround more user-friendly. Another option is to have our Fortran compiler knowledge-base separable from the rest of the package. scons could try to import them from, say, numpy_fcompilers first and then look inside numpy.distutils if numpy_fcompilers is not found. That way, a user could download a fresh knowledge-base into their source tree (and possibly tweak it) without the burden of installing a new numpy. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
Fernando Perez wrote: On Dec 4, 2007 12:27 PM, Robert Kern [EMAIL PROTECTED] wrote: user-friendly. Another option is to have our Fortran compiler knowledge-base separable from the rest of the package. scons could try to import them from, say, numpy_fcompilers first and then look inside numpy.distutils if numpy_fcompilers is not found. That way, a user could download a fresh knowledge-base into their source tree (and possibly tweak it) without the burden of installing a new numpy. Is this something that really needs to be a code package? Why can't this knowledge (or at least the easily overridable part of it) be packaged in one or more .conf/.ini plaintext files? In that way, users could easily grab new data files or tweak the builtin ones, and at build time say setup.py install --compiler_conf=~/my_tweaked.conf Is that impossible/unreasonable for some reason? It's not impossible, but there are at least a couple of places where it might be unreasonable. For example, look at the get_flags_arch() for Intel compilers: def get_flags_arch(self): v = self.get_version() opt = [] if cpu.has_fdiv_bug(): opt.append('-fdiv_check') if cpu.has_f00f_bug(): opt.append('-0f_check') if cpu.is_PentiumPro() or cpu.is_PentiumII() or cpu.is_PentiumIII(): opt.extend(['-tpp6']) elif cpu.is_PentiumM(): opt.extend(['-tpp7','-xB']) elif cpu.is_Pentium(): opt.append('-tpp5') elif cpu.is_PentiumIV() or cpu.is_Xeon(): opt.extend(['-tpp7','-xW']) if v and v = '7.1': if cpu.has_mmx() and (cpu.is_PentiumII() or cpu.is_PentiumIII()): opt.append('-xM') elif v and v = '8.0': if cpu.is_PentiumIII(): opt.append('-xK') if cpu.has_sse3(): opt.extend(['-xP']) elif cpu.is_PentiumIV(): opt.append('-xW') if cpu.has_sse2(): opt.append('-xN') elif cpu.is_PentiumM(): opt.extend(['-xB']) if (cpu.is_Xeon() or cpu.is_Core2() or cpu.is_Core2Extreme()) and cpu.getNCPUs()==2: opt.extend(['-xT']) if cpu.has_sse3() and (cpu.is_PentiumIV() or cpu.is_CoreDuo() or cpu.is_CoreSolo()): opt.extend(['-xP']) if cpu.has_sse2(): opt.append('-arch SSE2') elif cpu.has_sse(): opt.append('-arch SSE') return opt Expressing that without code could be hairy. That said, using configuration files as override mechanisms for each of the get_flags_*() methods would be feasible especially if there were a script to dump the current flag set to the configuration file. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
Fernando Perez wrote: On Dec 4, 2007 1:24 PM, Robert Kern [EMAIL PROTECTED] wrote: Fernando Perez wrote: Is this something that really needs to be a code package? Why can't this knowledge (or at least the easily overridable part of it) be packaged in one or more .conf/.ini plaintext files? In that way, users could easily grab new data files or tweak the builtin ones, and at build time say setup.py install --compiler_conf=~/my_tweaked.conf Is that impossible/unreasonable for some reason? It's not impossible, but there are at least a couple of places where it might be unreasonable. For example, look at the get_flags_arch() for Intel compilers: [...] I see. How about an alternate approach: exposing a simple api and allowing users to declare a *python* file to execfile() at load time looking for the config? Something like: setup.py install --compiler_conf=~/my_tweaked_config.py where the config file would be (sketch, not real code here): def make_flags(compiler, etc...): flags = [] return flags There could be a simple API for what functions the config file (down to their names and signatures) can declare, and if any of them are declared, they get called and their output is used. They get fed the default state of the same variables, so that they can choose to modify or outright replace them based on the user's need. The config code would then user_ns = {} execfile(user_config_filename,user_ns) for name,val in user_ns.items(): if name in approved_functions and callable(val): flags[name] = val(*approved_functions[name].default_args) What say you? Well, like I said, for tweaking, a simple data file works better than code. There's no need to do all of those if tests since I know what platform I'm on. We should definitely have a simple data file that we can read flags from. It's just the general case that requires code. One thing in favor of numpy_fcompilers is that we can ship the updates to the general case more frequently. This means that other packages using Fortran (but not tied to a particular platform) can ship the updated code instead of telling all of their users to read their Fortran compiler manuals or ask on the mailing list for the correct settings to work around our old defects. Tweaking should be the province of the developer and the desperate. We have to be able to tweak, but we should spend more time on preventing the need for tweaking. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview about how to do this? For example, the Intel Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make my edits there? Do I then add 'ifort' to the list in scons/core/default.py? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
David Cournapeau wrote: So to go back to your problem: if I understand correctly, what is needed is to update the scons tools. Since those are kept at one place, I think it would be safe to update them independently. But I don't understand exactly how this could be installed in the source tree without reinstalling numpy ? I think this is better than completely overriding the compilation flags, personally. But if you really want this possibility, I can add it, too, without too much trouble. I don't think you could install it into an already installed numpy package. My suggestion is to keep the implementations of the tools inside the numpy package as they are now *except* that we look for another package first before going inside numpy.distutils.scons.tools . I called it numpy_fcompilers though I now suspect numpy_scons_tools might be more appropriate. If the package numpy_scons_tools doesn't exist, the implementations inside numpy.distutils.scons.tools are used. A variation on this would be to provide an --option to the scons command to provide a package path to look for tools. E.g. python setup.py scons --tool-path=my_local_fcompilers,site_fcompilers,numpy.distutils.scons.tools This, too, is a workaround for the less-than-desirable situation of having numpy's sizable build infrastructure bundled with the numpy package itself. If this stuff were an entirely separate package focused on providing this scons-based build infrastructure, then we wouldn't have a problem. We could just update it on its own release schedule. People would probably be more willing to use the development versions of it, too, instead of having to also buy into the development version of their array package as well. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
David Cournapeau wrote: Robert Kern wrote: David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview about how to do this? For example, the Intel Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make my edits there? Do I then add 'ifort' to the list in scons/core/default.py? The basic rule is: if the code cannot run without a flag, the flag should be put in a tool, or at worse (but really if you have no choice) in numpyenv.py. If the flag is optimization, warning, etc... then it should be put into default.py. Basically, tools are not always up-to-date in scons, perticularly for fortran. So I provided a way to override the tools: as you noticed, you can put tools in .../scons/tools/, those will be picked up first. This is independent from adding ifort in scons/core/default.py. Right. In this case, -shared needs to be -dynamiclib on OS X, so this should go into the tool. For Mac OS X, you may be bitten by -undefined dynamic_lookup. This is my fault: this flag is added at the wrong place, I put it temporarily in the python extension builder, but this is not where it should be put. Depending on its meaning, I can put it at the right place: does it give the traditional unix semantic of enabling unresolved symbols instead of the default one, which is similar to windows (even for shared code, every symbol must be resolved) ? That's the basic idea. Rigorously, it's probably a bit more involved when you start considering two-level namespaces and framework. One thing to note is that this option is only valid for GNU compilers. Linking with ifort, I need to use -Wl,-undefined,dynamic_lookup . -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
David Cournapeau wrote: On Dec 5, 2007 1:14 PM, Robert Kern [EMAIL PROTECTED] wrote: This, too, is a workaround for the less-than-desirable situation of having numpy's sizable build infrastructure bundled with the numpy package itself. If this stuff were an entirely separate package focused on providing this scons-based build infrastructure, then we wouldn't have a problem. We could just update it on its own release schedule. People would probably be more willing to use the development versions of it, too, instead of having to also buy into the development version of their array package as well. The only problem I see is that this increases the chance of losing synchronization. I don't know if this is significant. The problem I see is that numpy-the-array-library and numpy.distutils-the-build-infrastructure are two related packages with *over*-synchronized cycles. We aren't going to push out a micro-release of numpy-the-array-library just because a new version of Intel Fortran comes out and changes its flags. IMHO, the only real solution would be to fix distutils (how many people want a shared library builder in python community, for example ?), but well, it is not gonna happen in a foreseeable future, unfortunately I don't see how that's relevant to the problem I raised. Supporting Fortran in the standard library would make the problem even worse. distutils is certainly not broken because it doesn't support Fortran. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
David Cournapeau wrote: On Dec 5, 2007 1:19 PM, Robert Kern [EMAIL PROTECTED] wrote: David Cournapeau wrote: Robert Kern wrote: David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview about how to do this? For example, the Intel Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make my edits there? Do I then add 'ifort' to the list in scons/core/default.py? The basic rule is: if the code cannot run without a flag, the flag should be put in a tool, or at worse (but really if you have no choice) in numpyenv.py. If the flag is optimization, warning, etc... then it should be put into default.py. Basically, tools are not always up-to-date in scons, perticularly for fortran. So I provided a way to override the tools: as you noticed, you can put tools in .../scons/tools/, those will be picked up first. This is independent from adding ifort in scons/core/default.py. Right. In this case, -shared needs to be -dynamiclib on OS X, so this should go into the tool. That's strange: -shared should not be used at all on mac os X. Either -bundle or -dynamiclib should be used (this is in applelink tool, so this is independant from the compiler used, normally). I was only reading code; I haven't tested building Fortran extensions, yet. However, using a generic link tool would be the wrong thing to do for most Fortran extensions, I think. Where does it get the correct Fortran runtime libraries from? Some Fortran compilers really like to be the linker when mixing languages. But I may have done something wrong, because I don't know much about mac os X idiosyncraties on this: basically, what's the difference between -dynamiclib and -bundle ? When I build python extension, I used the module scons builder, which is the same than shared library except on mac os X (on mac os X, shared libraries use -dynamiclib, modules use -bundle). I must confess that I used the thing which worked in thise case, without knowing exactly what i was doing. ifort only supports -dynamiclib. For the regular linker, -bundle is correct for building Python extensions; I may have to rethink about using ifort to link, then. Basically, a bundle can be dynamically loaded while dylibs can't, so Python uses bundles for extension modules. http://www.finkproject.org/doc/porting/shared.php What confuses me is that I successfully built some Fortran modules last night using numpy.distutils and ifort -dynamiclib. Hmm. One thing to note is that this option is only valid for GNU compilers. Linking with ifort, I need to use -Wl,-undefined,dynamic_lookup . Can't we just add a linker flag instead of using it from the compiler ? We still use the apple linker with ifort/icc, no ? I don't know. We'd have to locate all of the Fortran runtime libraries and add them. How do we do that? Or is that already done? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons
David Cournapeau wrote: On Dec 5, 2007 3:07 PM, Robert Kern [EMAIL PROTECTED] wrote: I don't see how that's relevant to the problem I raised. Supporting Fortran in the standard library would make the problem even worse. distutils is certainly not broken because it doesn't support Fortran. Fortran support is indeed certainly not in the scope of distutils. I was just answering to the general problem that we have a huge build infrastructure, not to the particular fortran problem. Having an infrastructure for adding easily new tools, that can be distributed separately, is something that distutils severely lacks IMHO. Ah, yes. I agree. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] site down...
Fernando Perez wrote: The whole scipy.org site seems down. Is it just on my end? No. Our new IT guy, Ryan Earl jre at enthought.com, is on the case. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] site down...
Steven H. Rogers wrote: Fernando Perez wrote: The whole scipy.org site seems down. Is it just on my end? Works for me, though it seems pretty slow. The system has been restarted. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multinomial question
Alan G Isaac wrote: I would think that multinomial(1,prob,size=ntrials).sum(axis=0) would be equivalent to multinomial(ntrials,prob) but the first gives a surprising result. (See below.) Explanation? A bug in rk_binomial_inversion(). Unfortunately, this looks like a logical bug in the sources I was deriving this code from. The safety bound on the search inversion search cuts out too early. Now that I re-examine it, it looks the bound (whichever of the multiple choice of bounds one could use) could always be legitimately exceeded, so there shouldn't be a bound at all. I'll have to dive deeper to figure out what is going on. This makes me grumpy. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multinomial question
Alan G Isaac wrote: I would think that multinomial(1,prob,size=ntrials).sum(axis=0) would be equivalent to multinomial(ntrials,prob) but the first gives a surprising result. (See below.) Explanation? Pretty much anyone who derives their binomial distribution algorithm from http://www.unc.edu/~gfish/fcmc.html is also wrong. SVN now has a bound such that CDF(bound) is within 1e-16 (or so) of 1.0. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Max on Complex Array
Tom Johnson wrote: What is the meaning of numpy's maximum on a complex-valued array? We impose a lexicographical ordering on the complex space. Complex numbers are compared first by their real component and then, if the real components are equal, by their imaginary component. In [1]: from numpy import * In [19]: c = random.randint(0, 5, 10) + random.random(10)*1j In [20]: c Out[20]: array([ 0.+0.68448275j, 1.+0.97849291j, 3.+0.22114928j, 4.+0.65409519j, 3.+0.91550523j, 4.+0.50667105j, 1.+0.34576644j, 4.+0.97286048j, 1.+0.07268317j, 0.+0.52885086j]) In [21]: c.sort() In [22]: c Out[22]: array([ 0.+0.52885086j, 0.+0.68448275j, 1.+0.07268317j, 1.+0.34576644j, 1.+0.97849291j, 3.+0.22114928j, 3.+0.91550523j, 4.+0.50667105j, 4.+0.65409519j, 4.+0.97286048j]) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
David M. Cooke wrote: On Dec 10, 2007, at 10:30 , Matthieu Brucher wrote: 2007/12/10, Alexander Michael [EMAIL PROTECTED]: On Dec 10, 2007 6:48 AM, David Cournapeau [EMAIL PROTECTED] wrote: Hi, Several people reported problems with numpy 1.0.4 (See #627 and #628, but also other problems mentionned on the ML, which I cannot find). They were all solved, as far as I know, by a binary I produced (simply using mingw + netlib BLAS/LAPACK, no ATLAS). Maybe it would be good to use those instead ? (I can recompile them if there is a special thing to do to build them) Do I understand correctly that you are suggesting removing ATLAS from the Windows distribution? Wouldn't this make numpy very slow? I know on RHEL5 I see a very large improvement between the basic BLAS/LAPACK and ATLAS. Perhaps we should make an alternative Windows binary available without ATLAS just for those having problems with ATLAS? That's why David proposed the netlib version of BLAS/LAPACK and not the default implementation in numpy. I would agree with David ;) Our versions of BLAS/LAPACK are f2c'd versions of the netlib 3.0 BLAS/ LAPACK (actually, of Debian's version of these -- they include several fixes that weren't upstream). So netlib's versions aren't going to be any faster, really. And netlib's BLAS is slow. Now, if there is a BLAS that's easier to compile than ATLAS on windows, that'd be improvement. The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Traceback on divide by zero error
Tony S Yu wrote: Hello, This is something that's been bothering for awhile. When numpy raises the following divide by zero error: Warning: divide by zero encountered in double_scalars is there a way to get a Traceback on where that warning occurred. In [1]: from numpy import * In [2]: seterr? Type: function Base Class: type 'function' Namespace:Interactive File: /Users/rkern/svn/numpy/numpy/core/numeric.py Definition: seterr(all=None, divide=None, over=None, under=None, invalid=None) Docstring: Set how floating-point errors are handled. Valid values for each type of error are the strings ignore, warn, raise, and call. Returns the old settings. If 'all' is specified, values that are not otherwise specified will be set to 'all', otherwise they will retain their old values. Note that operations on integer scalar types (such as int16) are handled like floating point, and are affected by these settings. Example: seterr(over='raise') # doctest: +SKIP {'over': 'ignore', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'} seterr(all='warn', over='raise') # doctest: +SKIP {'over': 'raise', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'} int16(32000) * int16(3) # doctest: +SKIP Traceback (most recent call last): File stdin, line 1, in ? FloatingPointError: overflow encountered in short_scalars seterr(all='ignore') # doctest: +SKIP {'over': 'ignore', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'} In [3]: seterr(divide='raise') Out[3]: {'divide': 'print', 'invalid': 'print', 'over': 'print', 'under': 'ignore'} In [4]: ones(10) / zeros(10) --- FloatingPointErrorTraceback (most recent call last) /Users/rkern/svn/mpl-toolkits/ipython console in module() FloatingPointError: divide by zero encountered in divide In [5]: seterr(all='ignore') Out[5]: {'divide': 'raise', 'invalid': 'print', 'over': 'print', 'under': 'ignore'} In [6]: ones(10) / zeros(10) Out[6]: array([ Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf]) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Overloading sqrt(5.5)*myvector
Gary Ruben wrote: Sorry this isn't an answer, just noise, but for those here who don't know, Bruce is the chief maintainer of the vpython project. I have found vpython aka the visual module to be a highly attractive and useful module for teaching physics. It would be great if someone with Boost experience would try to help him out. I wouldn't want him to get falsely disillusioned with this list as I for one have been looking forward to a fully numpy-compatible version of vpython. Please keep in mind that for many of us, this is the holiday season and we are on vacation. While I'm happy to check the list and give answers that are at the front of my head, deeper answers that require exploration or experimentation are beyond my available time. I'm sure others are in a similar situation. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] Doc-day
Matthieu Brucher wrote: Matthew B. will be working on converting SciPy tests to use nose per Fernando's email. If you are familiar with nose and want to help, please make sure to check with Matthew or Fernando first. I must have missed Fernando's email because I can't find the references for nose :( Look in SciPy Sprint Results. It's only a brief mention, though. What are its advantages against the current numpy.testing framework ? Primarily: * It is supported by someone else and gaining wide adoption by the rest of the Python community. Secondarily: * More flexible organization of tests. For nose, if it looks like a test, it's a test. numpy.testing collects test modules which are named like the module it is testing. E.g. for module.py = tests/test_module.py. * Test generators: def test_evens(): for i in range(0, 5): yield check_even, i, i*3 def check_even(n, nn): assert n % 2 == 0 or nn % 2 == 0 * Package- and module-level setup() and teardown() functions. * Test functions can be simple functions. They do not need to be organized into classes if you don't need classes. * Integrated doctest collection. * Detailed error/failure reporting. nose can print out the values of variables at the location of the error. * Integrated code coverage and profiling. * Dropping into pdb on errors and failures. * More flexible running of specific tests. E.g. when I'm working on getting a particular test function running, I can specify that exact test and not run the rest of the test suite. * Output capture. Tests can print out anything they like to be more informative, but they won't appear unless if the test fails. More thoroughly: http://somethingaboutorange.com/mrl/projects/nose/ -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array)
Andrew Straw wrote: Apologies if I've missed the discussion of this, but I was recently surprised by the following behavior (in svn trunk 4673). The following code runs without triggering the assertion. import numpy as np print np.__version__ a=np.int32(42) b=np.array([],dtype=np.int32) assert np.allclose(a,b) Is this expected behavior of numpy or is this a bug I should report? Bug, I think. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array)
Matthew Brett wrote: So, currently we have all and allclose giving the same answer: In [19]: a = array([]) In [20]: b = array([1]) In [21]: all(a == b) Out[21]: True In [22]: allclose(a, b) Out[22]: True Would we want the answers to be different? No. I wasn't thinking correctly, previously. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy code coverage
Stefan van der Walt wrote: Hi all, I read about Titus Brown's Figleaf code coverage tool [1] on the Planet SciPy aggregator [2]. The results of running figleaf on the numpy test-suite [3] covers Python code only. What the best way of discovering the C and C++ code coverage as well? I've never encountered any documents about checking C code coverage of Python extension modules. It is possible that the standard tools for C code coverage (e.g. gcov) would work fine with some finagling. If you try this, I would love to hear what you did. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] weird indexing
Mathew Yeates wrote: Hi Okay, here's a weird one. In Fortran you can specify the upper/lower bounds of an array e.g. REAL A(3:7) What would be the best way to translate this to a Numpy array? I would like to do something like A=numpy.zeros(shape=(5,)) and have the expression A[3] actually return A[0]. Or something. Any thoughts? The best way would depend on what you want to do with it. Why do you need this? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_FromAny does not accept a generator?
Neal Becker wrote: It seems that PyArray_FromAny does not accept a generator? Seems like this would be useful. It's difficult to do all the magical interpretation that PyArray_FromAny() does with a iterator of unknown length. In Python, we have fromiter() which will consume an iterator to make (only) a rank-1 array and thus sidestep a big chunk of the magical interpretation. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Nasty bug using pre-initialized arrays
Stuart Brorson wrote: On Fri, 4 Jan 2008, Stuart Brorson wrote: I just discovered this today. It looks like a bug to me. Please flame me mercilessly if I am wrong! :-) FWIW, here's what Matlab does: A = rand(1, 4) + rand(1, 4)*i A = Columns 1 through 3 0.7833 + 0.7942i 0.6808 + 0.0592i 0.4611 + 0.6029i Column 4 0.5678 + 0.0503i B = zeros(1, 4) B = 0 0 0 0 for idx=1:4; B(idx) = A(idx); end B B = Columns 1 through 3 0.7833 + 0.7942i 0.6808 + 0.0592i 0.4611 + 0.6029i Column 4 0.5678 + 0.0503i I realize NumPy != Matlab, but I'd wager that most users would think that this is the natural behavior.. Well, that behavior won't happen. We won't mutate the dtype of the array because of assignment. Matlab has copy(-on-write) semantics for things like slices while we have view semantics. We can't safely do the reallocation of memory [1]. [1] Well, we do have a .resize() method which will do the reallocation and raise an exception if there are views lingering about. However, this is only done when explicitly asked for because this is a feature that is useful in a limited number of situations. We will not allow it to be done implicitly. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Charles R Harris wrote: On Jan 4, 2008 2:05 PM, David Cournapeau [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: bzr is not tied to linux. It is, in that development is funded by Canonical, but I haven't used either on windows, so don't have any idea how they compare in that regard. In that sense, it's all pretty much a wash between the three. Selenic initially developed Mercurial in the aftermath of the Linux kernel Bitkeeper foofoorah, and they continue to use it to manage their kernel modules. If we want to talk about Windows support, we should stick to more concrete facts (like the availability of Windows shell integration, etc.) instead of vague inferences. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Nasty bug using pre-initialized arrays
Zachary Pincus wrote: Hello all, That's well and good. But NumPy should *never* automatically -- and silently -- chop the imaginary part off your complex array elements, particularly if you are just doing an innocent assignment! Doing something drastic like silently throwing half your data away can lead to all kinds of bugs in code written by somebody who is unaware of this behavior (i.e. most people)! It sounds to me like the right thing is to throw an exception instead of downcasting a data object. I'm not sure that I agree! I'd rather not have to litter my code with casting operations every time I wanted to down-convert data types (creating yet more temporary arrays...) via assignment. e.g.: A[i] = calculate(B).astype(A.dtype) vs. A[i] = calculate(B) Further, writing functions to operate on generic user-provided output arrays (or arrays of user-provided dtype; both of which are common e.g. in scipy.ndimage) becomes more bug-prone, as every assignment would need to be modified as above. This change would break a lot of the image-processing code I've written (where one often does computations in float or even double, and then re-scales and down-converts the result to integer for display), for example. I guess that this could be rolled in via the geterr/seterr mechanism, and could by default just print warnings. I agree that silent truncation can be troublesome, but not having to spell every conversion out in explicit ( and error-prone) detail is also pretty useful. (And arguably more pythonic.) There's some confusion in the conversation here. Tim already identified it, but since it's continuing, I'm going to reiterate. There are two related hierarchies of datatypes: different kinds (integer, floating point, complex floating point) and different precisions within a given kind (int8, int16, int32, int64). The term downcasting should probably be reserved for the latter only. It seems to me that Zach and Scott are defending downcasting of precisions within a given kind. It does not necessarily follow that the behavior we choose for dtypes within a given kind should be the behavior when we are dealing with dtypes across different kinds. We can keep the precision downcasting behavior that you want while raising an error when one attempts to assign a complex number into a floating point array. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Chris Barker wrote: hmmm. Everyone posting so far seems to be positive on this idea, but I'm not so sure. A few thoughts: 1) change is bad. It may be worth it, but this decision needs to be made very differently than if we were starting from scratch. 2) apparently svn merge sucks compared to other merge technology. svn (and cvs before it) is the only system I'm used, and yes, merging is painful, but I have to say that it appeared to be painful because it's a very hard problem. Can anyone comment on why these other systems seem so much better? Does it have anything to do with Centralized vs. Distributed at all? Tangentially, yes. DVCSes need to keep track of more information in order to be distributed. That information is extremely useful for managing merges properly. Centralized systems could track this information, but they don't *need* to in order to be functional, so they mostly haven't, yet. For each revision, the DVCS knows what revisions it derives from. SVN does not keep this information. SVN primarily just knows the text diffs from revision to revision. In particular, if I have a long-lived branch, I am going to merge in changes from the trunk while I'm working on it. When I go to merge the branch back into the trunk, I need to know which trunk-revisions I've already merged into the branch. SVN does not track this information. Tools like svnmerge.py track some of this information at the expense of some added clumsiness. It's worth noting that SVN 1.5 will be tracking such information. But that release is a long ways off. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to create an array of objects that are sequences?
Chris Barker wrote: Travis E. Oliphant wrote: I like the depth option, and that could be used with the current interface without too much difficulty, I think. This question does come up quite often and should probably be addressed. Yes, it has, but the depth option was discussed too, and while it might solve some problems, it would still be ambiguous: What if your objects were nested sequences, and you wanted to partly flatten them -- which would you flatten? I'm pretty sure that that is exactly the ambiguity that the depth option resolves. Can you give me an example where it's still ambiguous with the depth information provided? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Travis E. Oliphant wrote: I don't think it is time to move wholesale to something like Mercurial or bzr. I would prefer it if all of the Enthought-hosted projects moved to the (new) system at once, which is not going to happen in the short term (but long term of course it's an open question). I think that's irrelevant. There is absolutely no reason that we should force all of the Enthought-hosted projects to move in sync. We would have reason if we were being asked to host a different centralized VCS with a complicated server, but hosting Mercurial or bzr is nearly trivial. We already do it for me: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi The remaining thing we would have to support is the Trac integration. While not as trivial as simply hosting the repositories, it's not a very large commitment. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to create an array of objects that are sequences?
Christopher Barker wrote: Robert Kern wrote: Chris Barker wrote: What if your objects were nested sequences, and you wanted to partly flatten them -- which would you flatten? I'm pretty sure that that is exactly the ambiguity that the depth option resolves. Can you give me an example where it's still ambiguous with the depth information provided? I was imagining: Say you have a bunch of nested sequences that, fully expanded, would yield a (i,j,k) rank-3 array. If you wanted a rank-2 array, you could have either: a (i,j) array with each element being a length-k sequence or: a (i,k) array, with each element being a length-j sequence. This is quite trivial with numpy n-d arrays. However, while you could build either of those from nested sequences, there isn't, in fact, and object in there that is that length-j sequence. And I think that's the key difference. I don't think array() should be responsible for *arbitrarily* flattening nested sequences in order to reinterpret structure. Instead, I think we really only need array() to be able to answer the question, is this object an atom or a sequence I need to descend into? Essentially, for all valid index-tuples (say, [i,j,k]) in the array: arr[i,j,k] == original[i,j,k] I guess this really points to the power of n-d arrays! Yes. Since one can get the (i,k) array after getting the (i,j,k) array and reshaping, I don't think we need to support it in array(). -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Bill Baxter wrote: http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram This is a bit puzzling. I understand better merging isn't the only reason to choose DVCS, but the above page basically says that Mercurial just uses whatever external merge program it can find. So the file-level merging sounds like it must really be no different from other VCSs. So it is really just proper merging of directory renames and the like that make it superior? No. If you'll pardon my repeating myself: DVCSes need to keep track of more information in order to be distributed. That information is extremely useful for managing merges properly. Centralized systems could track this information, but they don't *need* to in order to be functional, so they mostly haven't, yet. For each revision, the DVCS knows what revisions it derives from. SVN does not keep this information. SVN primarily just knows the text diffs from revision to revision. In particular, if I have a long-lived branch, I am going to merge in changes from the trunk while I'm working on it. When I go to merge the branch back into the trunk, I need to know which trunk-revisions I've already merged into the branch. SVN does not track this information. Tools like svnmerge.py track some of this information at the expense of some added clumsiness. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
David Cournapeau wrote: Does good merging only depends on the above ? Martin Pool, one of the bzr programmer, wrote this article two years ago: http://sourcefrog.net/weblog/software/vc/derivatives.html which I found both enlightening and easy to follow. My terminology was fuzzy/incorrect. By revision, I meant changeset rather than snapshot. The main thrust of that article is about the value of viewing VCS history as a sequence of changesets rather than a sequence of snapshots. FWIW, svnmerge and SVN 1.5 Merge Tracking are changeset-oriented rather than snapshot-oriented. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Bill Baxter wrote: On Jan 6, 2008 6:38 PM, Robert Kern [EMAIL PROTECTED] wrote: Bill Baxter wrote: http://www.selenic.com/mercurial/wiki/index.cgi/MergeProgram This is a bit puzzling. I understand better merging isn't the only reason to choose DVCS, but the above page basically says that Mercurial just uses whatever external merge program it can find. So the file-level merging sounds like it must really be no different from other VCSs. So it is really just proper merging of directory renames and the like that make it superior? No. If you'll pardon my repeating myself: DVCSes need to keep track of more information in order to be distributed. That information is extremely useful for managing merges properly. Centralized systems could track this information, but they don't *need* to in order to be functional, so they mostly haven't, yet. For each revision, the DVCS knows what revisions it derives from. SVN does not keep this information. SVN primarily just knows the text diffs from revision to revision. In particular, if I have a long-lived branch, I am going to merge in changes from the trunk while I'm working on it. When I go to merge the branch back into the trunk, I need to know which trunk-revisions I've already merged into the branch. SVN does not track this information. Tools like svnmerge.py track some of this information at the expense of some added clumsiness. Ok. Sorry for not reading that closer. So what you're saying is that the magic is in the deciding of exactly which revisions of which files to run the merge program on? That's the main idea. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] parallel numpy (by Brian Granger) - any info?
dmitrey wrote: The only one thing I'm very interested in for now - why the most simplest matrix operations are not implemented to be parallel in numpy yet (for several-CPU computers, like my AMD Athlon X2). First of all it's related to matrix multiplication and devision, either point or matrix (i.e. like A\B, A*B, dot(A,B)). Eric Jones has made an attempt. http://svn.scipy.org/svn/numpy/branches/multicore/ Unfortunately, the overhead of starting the threads and acquiring/releasing thread locks wipes out most of the performance gains until you get fairly large arrays. It is possible that this comes from the particular implementation, rather than being intrinsic to the problem. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Using svnmerge on numpy: am I missing something ?
David Cournapeau wrote: Matthieu Brucher wrote: Oups, safe for the /trunk:1-2871 part. This should be deleted before a commit to the trunk, I think. Yes, that's what I (quite unclearly) meant: since revision numbers are per- repository in svn, I don't understand the point of tracking trunk revisions: I would think that tracking the last merged version for each branch to be enough (for the kind of merge svn does, at least). If trunk version are tracked, then I would expect two branches using svnmerge to clash each other, In fact, the trunk should be tracked from all the branches, although there will be the problem with merging the different branches (I did not have many troubles with that, but I only tried with a few differences) into the trunk. I don't think only one branch wants to be up to date with the trunk ;). Sure, that's what I was trying to do (tracking the trunk). But if svnmerge needs the information from the trunk, this means each branch will need a different value, whereas only one value is possible... IIRC, its presence in the trunk/'s svnmerge-integrated property is a mistake. It belongs in each of the branches independently. For some reason, it got merged into the trunk from one of the branches by mistake, possibly because someone started merge tracking with svnmerge then another used svn merge. It should be removed. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] def of var of complex
Neal Becker wrote: I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: var (real part) - (close to 1.0) var (imag part) - (close to 1.0) but var (complex array) - (close to complex 0) Is that not a strange definition? There is some discussion on this in the tracker. http://projects.scipy.org/scipy/numpy/ticket/638 The current state of affairs is that the implementation of var() just naively applies the standard formula for real numbers. mean((x - mean(x)) ** 2) I think this is pretty obviously wrong prima facie. AFAIK, no one considers this a valid definition of variance for complex RVs or in fact a useful value. I think we should change this. Unfortunately, there is no single alternative but several. 1. Punt. Complex numbers are inherently multidimensional, and a single scale parameter doesn't really describe most distributions of complex numbers. Instead, you need a real covariance matrix which you can get with cov([z.real, z.imag]). This estimates the covariance matrix of a 2-D Gaussian distribution over RR^2 (interpreted as CC). 2. Take a slightly less naive formula for the variance which seems to show up in some texts: mean(absolute(z - mean(z)) ** 2) This estimates the single parameter of a circular Gaussian over RR^2 (interpreted as CC). It is also the trace of the covariance matrix above. 3. Take the variances of the real and imaginary components independently. This is equivalent to taking the diagonal of the covariance matrix above. This wouldn't be the definition of *the* complex variance that anyone else uses, but rather another form of punting. There isn't a single complex variance to give you, but in the spirit of broadcasting, we'll compute the marginal variances of each dimension independently. Personally, I like 1 a lot. I'm hesitant to support 2 until I've seen an actual application of that definition. The references I have been given in the ticket comments are all early parts of books where the authors are laying out definitions without applications. Personally, it feels to me like the authors are just sticking in the absolute()'s ex post facto just so they can extend the definition they already have to complex numbers. I'm also not a fan of the expectation-centric treatments of random variables. IMO, the variance of an arbitrary RV isn't an especially important quantity. It's a parameter of a Gaussian distribution, and in this case, I see no reason to favor circular Gaussians in CC over general ones. But if someone shows me an actual application of the definition, I can amend my view. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] def of var of complex
Charles R Harris wrote: Suppose you have a set of z_i and want to choose z to minimize the average square error $ \sum_i |z_i - z|^2 $. The solution is that $z=\mean{z_i}$ and the resulting average error is given by 2). Note that I didn't mention Gaussians anywhere. No distribution is needed to justify the argument, just the idea of minimizing the squared distance. Leaving out the ^2 would yield another metric, or one could ask for a minmax solution. It is a question of the distance function, not probability. Anyway, that is one justification for the approach in 2) and it is one that makes a lot of applied math simple. Whether of not a least squares fit is useful is different question. If you're not doing probability, then what are you using var() for? I can accept that the quantity is meaningful for your problem, but I'm not convinced it's a variance. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] def of var of complex
Neal Becker wrote: 2 is what I expected. Suppose I have a complex signal x, with additive Gaussian noise (i.i.d, real and imag are independent). y = x + n Not only do the real and imag marginal distributions have to be independent, but also of the same scale, i.e. Re(n) ~ Gaussian(0, sigma) and Im(n) ~ Gaussian(0, sigma) for the same sigma. Consider an estimate \hat{x} = y. What is the mean-squared-error E[(y - x)^2] ? Definition 2 is consistent with that, and gets my vote. Ah, you have to be careful. What you wrote is what is implemented. Definition 2 is consistent with this, instead: E[|y - x|^2] But like I said, I see no particular reason to favor circular Gaussians over the general form for the implementation of numpy.var(). -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Does float16 exist?
Charles R Harris wrote: I see that there are already a number of parsers available for Python, SPARK, for instance is included in the 2.5.1 distribution. No, it isn't. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] def of var of complex
Charles R Harris wrote: On Jan 8, 2008 7:48 PM, Robert Kern [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Charles R Harris wrote: Suppose you have a set of z_i and want to choose z to minimize the average square error $ \sum_i |z_i - z|^2 $. The solution is that $z=\mean{z_i}$ and the resulting average error is given by 2). Note that I didn't mention Gaussians anywhere. No distribution is needed to justify the argument, just the idea of minimizing the squared distance. Leaving out the ^2 would yield another metric, or one could ask for a minmax solution. It is a question of the distance function, not probability. Anyway, that is one justification for the approach in 2) and it is one that makes a lot of applied math simple. Whether of not a least squares fit is useful is different question. If you're not doing probability, then what are you using var() for? I can accept that the quantity is meaningful for your problem, but I'm not convinced it's a variance. Lots of fits don't involve probability distributions. For instance, one might want to fit a polynomial to a mathematical curve. This sort of distinction between probability and distance goes back to Gauss himself, although not in his original work on least squares. Whether or not variance implies probability is a semantic question. Well, the problem in front of us is entirely semantics: What does the string var(z) mean? Are we going to choose an mechanistic definition: var(z) is implemented in such and such a way and interpretations are left open? In that case, why are we using the string var(z) rather than something else? We're also still left with the question as to which such and such implementation to use. Alternatively, we can look at what people call variances and try to implement the calculation of such. In that case, the term variance tends to crop up (and in my experience *only* crop up) in statistics and probability. Certain implementations of the calculations of such quantities have cognates elsewhere, but those cognates are not themselves called variances. My question to you is, is the resulting average error a variance? I.e., do people call it a variance outside of SP? There are any number of computations that are useful but are not variances, and I don't think we should make var(z) implement them. In SP, the single quantity variance is well defined for real RVs, even if you step away from Gaussians. It's the second central moment of the PDF of the RV. When you move up to CC (or RR^2), the definition of moment changes. It's no longer a real number or even a scalar; the second central moment is a covariance matrix. If we're going to call something the variance, that's it. The circularly symmetric forms are special cases. Although option #2 is a useful quantity to calculate in some circumstances, I think it's bogus to give it a special status. I think if we are going to compute a single number, 2) is as good as anything even if it doesn't capture the shape of the scatter plot. A 2D covariance wouldn't necessarily capture the shape either. True, but it is clear exactly what it is. The function is named cov(), and it computes covariances. It's not called shape_of_2D_pdf(). Whether or not one ought to compute a covariance is not cov()'s problem. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] def of var of complex
Travis E. Oliphant wrote: Robert Kern wrote: Neal Becker wrote: I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: var (real part) - (close to 1.0) var (imag part) - (close to 1.0) but var (complex array) - (close to complex 0) Is that not a strange definition? 2. Take a slightly less naive formula for the variance which seems to show up in some texts: mean(absolute(z - mean(z)) ** 2) This estimates the single parameter of a circular Gaussian over RR^2 (interpreted as CC). It is also the trace of the covariance matrix above. I tend to favor this interpretation because it is used quite heavily in signal processing applications where circular Gaussian random variables show up quite a bit --- so much so, in fact, that most EE folks would expect this as the output and you would have to explain to them why there may be other choices that make sense. So, #2 is kind of a nod to the signal-processing community (especially the communication section). sigh Fair enough. I relent. You implement it; I'll document the correct^Wcov() alternative. :-) But, there is also merit to me in #3 (although it may be harder to explain why the variance returns a complex number --- if that is what you meant). Yes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Template system
Charles R Harris wrote: Hi All, I'm thinking of adding the template system I pulled out of Django to Numpy to use in place of the current code generator. Its advantages are documentation, flexibility, and maintainance. The code totals about 470 KB, comes with a BSD license, and is compatible with Python = 2.3. I want to bring this up on the list because I'm sure there will be objections along the lines of why do we need that? So I want to see how strenuous the objections are. It's too big. If you want a better template system than the current one, there are a few which are contained in a single file. That would be fine to include, but I object strongly to incorporating Jinja. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Template system
Charles R Harris wrote: On Jan 11, 2008 2:58 PM, Robert Kern [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Charles R Harris wrote: Hi All, I'm thinking of adding the template system I pulled out of Django to Numpy to use in place of the current code generator. Its advantages are documentation, flexibility, and maintainance. The code totals about 470 KB, comes with a BSD license, and is compatible with Python = 2.3. I want to bring this up on the list because I'm sure there will be objections along the lines of why do we need that? So I want to see how strenuous the objections are. It's too big. If you want a better template system than the current one, there are a few which are contained in a single file. That would be fine to include, but I object strongly to incorporating Jinja. Ummm, harsh. I'm certainly open to suggestions. I took a closer look at the django system and reduced it to 308K, 70K zipped. There was a lot of space taken up by the .svn directories inherited from django and *.pyc files. I could probably knock another 50K off by removing some functionality we don't really need: html, http, dates, time, and maybe template inheritance. Just for comparison, here are some things that are in a fresh checkout of numpy, including the .svn files Numpy 23 MB f2py 5.4 MB doc/swig 1.5 MB code_generators 192K So adding the template subsystem would increase the repository size by about 1.4%. Do you have any objections in addition to the size? Yes. I'm not convinced that we need anything better than we have now. Show me how using Jinja templates makes the current code better. Check the one-file templating modules (like YAPTU[1] or Cog[2]) out there to see if those are sufficient. Jinja and its kin (Mako, Cheetah, etc.) are intended for a very different use case, and their design, implementation, and footprint are optimized for that. [1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52305 [2] http://nedbatchelder.com/code/cog/ -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Template system
Charles R Harris wrote: OK, so far I've knocked it down to 35KB by removing stuff I'm not interested in. It is now smaller than Cog, and 7x larger than the file we now use to do the same job. I'm pretty sure I can make it leaner than that. It remains extensible. Can you put a tarball up somewhere? Django generally requires environment variables to configure itself. There is a way to configure these from code, but it looks like that pulls in django.conf. This is my primary source of discomfort with using Django templates. My experience with using Django components outside of a Django web app has been frustrating. The more modifications we make, the more we have to maintain. And if any of it causes problems for someone trying to build numpy, that's more questions we need to answer. I want to answer numpy questions, not Django questions. I would be much, much more comfortable with an implementation that can simply be dropped in without modification. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] casting
Neal Becker wrote: numpy frequently refers to 'casting'. I'm not sure if that term is ever defined. I believe it has the same meaning as in C. In that case, it is unfortunately used to mean 2 different things. There are casts that do not change the underlying bits (such as a pointer cast), and there are casts that actually convert to different bits (such as float - double). I think numpy means the latter. When an array where the underlying data is one type, a cast to another type means actually reallocating and converting the data. Yes, that is usually what people mean when they use _casting_ in the context of numpy. It is the more frequently performed operation of the two. The former can be accomplished with the .view(dtype) method of ndarrays. It often occurs that I have an algorithm that can take any integral type, because it is written with c++ templates. In that case, I don't want to use PyArray_FROMANY, because I don't want to unecessarily convert the array data. Instead, I'd like to inquire what is the preferred type of the data. The solution I'm exploring is to use a function I call 'preferred_array_type'. This uses the __array_struct__ interface to find the native data type. I chose to use this interface, because then it will work with both numpy arrays and other array-like types. Any thoughts on all of this? I'm not sure what you mean by preferred type of the data. Do you mean the dtype of the array as it comes in? There are several functions and function macros in the numpy C API which take differing amounts of information. For example, * PyArray_FROM_O(PyObject*onj) just takes an object. * PyArray_FROM_OF(PyObject* obj, int req) takes an object and flags like NPY_CONTIGUOUS. * PyArray_FROM_OT(PyObject* obj, int typenum) takes an object and a type number. * PyArray_FROM_OTF(PyObject* obj, int typenum, int req) takes an object, type, and flags. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] casting
Neal Becker wrote: Robert Kern wrote: Neal Becker wrote: numpy frequently refers to 'casting'. I'm not sure if that term is ever defined. I believe it has the same meaning as in C. In that case, it is unfortunately used to mean 2 different things. There are casts that do not change the underlying bits (such as a pointer cast), and there are casts that actually convert to different bits (such as float - double). I think numpy means the latter. When an array where the underlying data is one type, a cast to another type means actually reallocating and converting the data. Yes, that is usually what people mean when they use _casting_ in the context of numpy. It is the more frequently performed operation of the two. The former can be accomplished with the .view(dtype) method of ndarrays. It often occurs that I have an algorithm that can take any integral type, because it is written with c++ templates. In that case, I don't want to use PyArray_FROMANY, because I don't want to unecessarily convert the array data. Instead, I'd like to inquire what is the preferred type of the data. The solution I'm exploring is to use a function I call 'preferred_array_type'. This uses the __array_struct__ interface to find the native data type. I chose to use this interface, because then it will work with both numpy arrays and other array-like types. Any thoughts on all of this? I'm not sure what you mean by preferred type of the data. Do you mean the dtype of the array as it comes in? There are several functions and function macros in the numpy C API which take differing amounts of information. For example, * PyArray_FROM_O(PyObject*onj) just takes an object. * PyArray_FROM_OF(PyObject* obj, int req) takes an object and flags like NPY_CONTIGUOUS. * PyArray_FROM_OT(PyObject* obj, int typenum) takes an object and a type number. * PyArray_FROM_OTF(PyObject* obj, int typenum, int req) takes an object, type, and flags. Let me try again to explain. I don't want to convert to some type first - that would be a waste. I need to find out what is the native data type of the input array first. Also, I'd like to allow that the input is not a PyArray, but could be something conforming to __array_struct__ interface. So, I need to find the native data type _first_, _then_ call the appropriate PyArray_FROM... I'm sorry, I still think we're talking past each other. What do you mean by native data type? If you just want to get an ndarray without specifying a type, use PyArray_FROM_O(). That's what it's for. You don't need to know the data type beforehand. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] RFC: out of range slice indexes
Neal Becker wrote: I've never liked that python silently ignores slices with out of range indexes. I believe this is a source of bugs (it has been for me). It goes completely counter to the python philosophy. I vote to ban them from numpy. from numpy import array x = array (xrange (10)) x[11] Traceback (most recent call last): File stdin, line 1, in module IndexError: index out of bounds x[:12] = 2 x array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) len (x) 10 Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to _extend_ x to have lenght 12, at least _that_ would be reasonable (but I'm not advocating that position). I believe that out of bounds indexes should always throw IndexError. We can't change that in Python now, but maybe we can in numpy. -1. Regardless of the merits if we had a blank slate, there is code that depends on this, specifically my code. It simplifies certain operations that would otherwise need tedious special case-handling. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] RFC: out of range slice indexes
Neal Becker wrote: Robert Kern wrote: Neal Becker wrote: I've never liked that python silently ignores slices with out of range indexes. I believe this is a source of bugs (it has been for me). It goes completely counter to the python philosophy. I vote to ban them from numpy. from numpy import array x = array (xrange (10)) x[11] Traceback (most recent call last): File stdin, line 1, in module IndexError: index out of bounds x[:12] = 2 x array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) len (x) 10 Silently ignoring the error x[:12] is a bad idea, IMO. If it meant to _extend_ x to have lenght 12, at least _that_ would be reasonable (but I'm not advocating that position). I believe that out of bounds indexes should always throw IndexError. We can't change that in Python now, but maybe we can in numpy. -1. Regardless of the merits if we had a blank slate, there is code that depends on this, specifically my code. It simplifies certain operations that would otherwise need tedious special case-handling. For example? def ichunk(arr, chunk_size=10): for i in range(0, len(arr), chunk_size): yield arr[i:i+chunk_size] -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] casting
Neal Becker wrote: Jon Wright wrote: I'm sorry, I still think we're talking past each other. What do you mean by native data type? If you just want to get an ndarray without specifying a type, use PyArray_FROM_O(). That's what it's for. You don't need to know the data type beforehand. What I have wanted in the past (and what I thought Neal was after) is a way to choose which function to call according to the typecode of the data as it is currently in memory. I don't want to convert (or cast or even touch the data) but just call a type specific function instead. C++ templates can take some of the tedium out of that, but in some cases algorithms may be different too. Guessing which sort algorithm to use springs to mind. Rather than saying give me the right kind of array, I think there is an interest in saying choose which function is the best for this data. Something like: PyArrayObject* array = PyArray_FROM_O( (PyObject*) O ); type = array - descr - type_num ; switch (type){ caseNPY_BYTE : signed_func(array); caseNPY_UBYTE : unsigned_func(array); // etc It sort of implies having a C++ type hierarchy for numpy arrays and casting array to be a PyFloatArray or PyDoubleArray etc? The extra confusion might be due to the way arrays can be laid out in memory - indexing into array slices is not always obvious. Also if you want to make sure your inner loop goes over the fast index you might want an algorithm which reads the strides when it runs. Sorry if I've only added to the confusion. Cheers, Jon This is close to what I'm doing. If I really can handle any type, then FROM_O is fine. Commonly, it's a little more complicated. Here is an example, in pseudo-code (real code is in c++): if (native_type_of (x) is int or long): do_something_with (convert_to_long_array(x)) elif (native_type_of (x) is complex): do_something_with (convert_to_complex_array (x)) In the above, native_type_of means: if (hasattr (x, __array_struct__)): array_if = PyCObject_AsVoidPtr (x.__array_struct__) return (map_array_if_to_typenum (array_if)) So this means for any numpy array or any type supporting __array_struct__ protocol, find out the native data type. I don't want to use FROM_O here, because I really can only handle certain types. If I used FROM_O, then after calling FROM_O, if the type was not one I could handle, I'd have to call FromAny and convert it. Or, in the case above, if I was given an array of int32 which I'd want to handle as long (int64), I'd have to convert it again. Okay, I think I see now. I'm not sure what numpy could do to make your code more elegant. I would recommend just using PyArray_FROM_O() or PyArray_EnsureArray() to get a real ndarray object. They should not copy when the object is already an ndarray or if it satisfies the array interface; however, it will also handle the cases when you have a Python-level __array_interface__ or just nested sequences, too. You can look at the PyArray_Descr directly, dispatch the types that you can handle directly, and then convert for the types which you can't. You will not get any extraneous conversions or copies of data. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy quit working
Kevin Christman wrote: I've started to use numpy (1.0.4) and matplotlib but it suddenly quit working. Now when I import a simple test script: import numpy numpy.linspace(0,1) it gives the following error: Traceback (most recent call last): File D:/anothertest.py, line 1, in module import numpy File C:\Python25\lib\site-packages\numpy\__init__.py, line 37, in module import testing File C:\Python25\lib\site-packages\numpy\testing\__init__.py, line 3, in module from numpytest import * File C:\Python25\lib\site-packages\numpy\testing\numpytest.py, line 19, in module from numpy.distutils.exec_command import splitcmdline File C:\Python25\lib\site-packages\numpy\distutils\__init__.py, line 6, in module import ccompiler File C:\Python25\lib\site-packages\numpy\distutils\ccompiler.py, line 46, in module replace_method(CCompiler, 'spawn', CCompiler_spawn) File C:\Python25\lib\site-packages\numpy\distutils\ccompiler.py, line 24, in replace_method m = new.instancemethod(func, None, klass) AttributeError: 'module' object has no attribute 'instancemethod' However, other import statements, like import time work ok. The problem first started when a script would only run if on the root of the drive (e.g. D:\), but now even if I run it from there it doesn't work. What's going on? Do you have a module called new.py somewhere? If so, then it is overriding the standard library's new module which we use. import new new module 'new' from 'C:\Python25\lib\new.pyc' If that filename does not point to 'C:\Python25\lib\new.pyc', then you have a problem. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] def of var of complex
Neal Becker wrote: Neal Becker wrote: I noticed that if I generate complex rv i.i.d. with var=1, that numpy says: var (real part) - (close to 1.0) var (imag part) - (close to 1.0) but var (complex array) - (close to complex 0) Is that not a strange definition? I don't think there is any ambiguity about the definition of the variance of complex. Var(x) = E{(x-E[x])^2} = E{x}^2 - E{x} That's currently what's implemented, but there is simply no evidence that anyone ever uses this definition for complex random variables. Note that for real variables, E{(x-E[x])^2} = E{|x-E[x]|^2} but for complex variables, there is a large difference. Since the || are superfluous with real variables, probability texts rarely include them unless if they are then going on to talk about complex variables. If you want to extend the definition for real variables to complex variables, that is an ambiguity you have to resolve. There is, apparently, a large body of statistical signal processing literature that defines the complex variance as E{|z-E[z]|^2} so, I (now) believe that this is what should be implemented. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Docstring page, out of date?
David Cournapeau wrote: Nobody likes to write doc, specially if you have to follow many rules. Not being able to see the result does not help. I don't know much about the current documation tools situation with python, but I have seen good, automatically generated doc for python modules: for example, the current developement doc of python (http://docs.python.org/dev/) looks pretty good to me, and using the same tools as python makes more sense than forking our own, no ? Or am I missing something fundamental ? Yes. Sphinx does not do automatically generated documentation. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Building a static libnumpy
Jussi Enkovaara wrote: Hi, I need to use numpy in an environment which does not support shared libraries. Previously, I have used the old Numeric where I created a Makefile to build a static Numeric library which was later on linked to the python interpreter. As far as I understood, numpy uses sort of extended distutils. I was wondering if this extended distutils enables building of static libraries or do I have to go the cumbersome Makefile-way again? Cumbersome Makefile, sorry. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Docstring standard: how to specify variable types
Stefan van der Walt wrote: Hi Charles On Wed, Jan 23, 2008 at 09:34:44AM -0700, Charles R Harris wrote: 2. How do we specify default values? I like to put them first in the list: {-1, integer} When exactly is this list used? That should be made clear in the standard as well. No special processing is done. I omit it. If I need to talk about the default value, I do it in prose in the argument's description. Note that the default value is not always the same thing as the value in the signature, particularly when kwd=None. But in those cases, I usually need a sentence to describe what's happening, so that's where I always put that information. 3. Why do we need the optional keyword (the function signature already indicates that the parameter is optional). It's extra information, true, but that isn't always a bad thing. It's like looking up whole numbers in a book index and, instead of the page reference, the index directs you to numbers, whole. Flip, flip, flip. Drives me crazy. Besides, the function signature might be missing. When would the function signature be missing? In C functions we copy the signature into the docstring. I am concerned about duplicating information that may change. 4. Do we really need the Other Parameters list? It would make more sense to split positional and keyword arguments, but I'm not even sure that is necessary, since that information is already specified in the function signature. I agree, but Travis likes this section and I don't feel strongly about it. If I understood its role properly, I would be happier to use it, but at the moment I'm not convinced that it contributes anything useful. Parameters are normally sorted from most to least valuable anyway. Look at the scipy.optimize.fmin*() functions, for instance. If you don't feel like using it, don't. None of the numpy function signatures should be long enough to need it. 5. Is the {'hi', 'ho'} syntax used when a parameter can only assume a limited number of values? In Python {} is a dictionary, so why not use ('hi','ho') instead? Either would be fine. IIRC, the {} was inherited from epydoc consolidated fields. I thought that, since we threw all epydoc guidelines out the window, this would be a good time talk about it -- the next docday is just around the corner! Personally, I don't think it's worth standardizing. If it's readable and valid reST, just do it. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numscons 0.3.0 release
dmitrey wrote: Hi all, I don't know much about what are these scons are, if it's something essential (as it seems to be from amount of mailing list traffic) why can't it be just merged to numpy, w/o making any additional branches? It's a very large, still experimental change to the entire build infrastructure. It requires branches for us to evaluate. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] difference numpy/matlab
Stuart Brorson wrote: I have to agree with Lorenzo. There is no natural ordering of the complex numbers. Any way you order them is arbitrary. Accepting this, the question then becomes what should NumPy do when the user tries to do order comparison operations on complex numbers. The problem is that NumPy is schizophrenic. Watch this: -- session log - In [20]: A = numpy.array([3+1j, 1+3j, -1-3j, -1+3j, -3-1j]) In [21]: B = A[numpy.random.permutation(5)] In [22]: In [22]: A Out[22]: array([ 3.+1.j, 1.+3.j, -1.-3.j, -1.+3.j, -3.-1.j]) In [23]: B Out[23]: array([-1.+3.j, 3.+1.j, -1.-3.j, 1.+3.j, -3.-1.j]) In [24]: numpy.greater(A, B) Out[24]: array([ True, False, False, False, False], dtype=bool) In [25]: numpy.maximum(A, B) Out[25]: array([ 3.+1.j, 3.+1.j, -1.-3.j, 1.+3.j, -3.-1.j]) In [26]: In [26]: 3+1j -1+3j --- type 'exceptions.TypeError' Traceback (most recent call last) /tmp/test/web/ipython console in module() type 'exceptions.TypeError': no ordering relation is defined for complex numbers /session log -- No, numpy is entirely consistent. In[26] has Python's complex numbers, not numpy's. In [1]: from numpy import complex64 In [2]: complex64(3+1j) complex64(-1+3j) Out[2]: True -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
Pearu Peterson wrote: On Fri, February 1, 2008 1:28 pm, Andrea Gavana wrote: Hi All, I sent a couple of messages to f2py mailing list, but it seems like my problem has no simple solution so I thought to ask for some suggestions here. Sorry, I haven't been around there long time. Are you going to continue not reading the f2py list? If so, you should point everyone there to this list and close the list. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Unexpected behavior with numpy array
Damian Eads wrote: Here's another question: is there any way to construct a numpy array and specify the buffer address where it should store its values? I ask because I would like to construct numpy arrays that work on buffers that come from mmap. Can you clarify that a little? By buffer do you mean a Python buffer() object? By mmap do you mean Python's mmap in the standard library? numpy has a memmap class which subclasses ndarray to wrap a mmapped file. It handles the opening and mmapping of the file itself, but it could be subclassed to override this behavior to take an already opened mmap object. In general, if you have a buffer() object, you can make an array from it using numpy.frombuffer(). This will be a standard ndarray and won't have the conveniences of syncing to disk that the memmap class provides. If you don't have a buffer() object, but just have a pointer allocated from some C code, then you *could* fake an object which exposes the __array_interface__() method to describe the memory. The numpy.asarray() constructor will use that to make an ndarray object that uses the specified memory. This is advanced stuff and difficult to get right because of memory ownership and object lifetime issues. If you can modify the C code, it might be easier for you to have numpy allocate the memory, then make the C code use that pointer to do its operations. But look at numpy.memmap first and see if it fits your needs. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Unexpected behavior with numpy array
Damian Eads wrote: Robert Kern wrote: Damian Eads wrote: Here's another question: is there any way to construct a numpy array and specify the buffer address where it should store its values? I ask because I would like to construct numpy arrays that work on buffers that come from mmap. Can you clarify that a little? By buffer do you mean a Python buffer() object? Yes, I mean the .data field of a numpy array, which is a buffer object, and points to the memory where an array's values are stored. Actually, the .data field is always constructed by ndarray; it is never provided *to* ndarray even if you construct the ndarray from a buffer object. The buffer object's information is interpreted to construct the ndarray object and then the original buffer object is ignored. The .data attribute will be constructed on-the-fly when it is requested. In [9]: from numpy import * In [10]: s = '' In [11]: b = buffer(s) In [12]: a = frombuffer(b, dtype=int32) In [13]: a.data is b Out[13]: False In [14]: d1 = a.data In [15]: d2 = a.data In [16]: d1 is d2 Out[16]: False By mmap do you mean Python's mmap in the standard library? I actually was referring to the C Standard Library's mmap. My intention was to use a pointer returned by C-mmap as the .data buffer to store array values. numpy has a memmap class which subclasses ndarray to wrap a mmapped file. It handles the opening and mmapping of the file itself, but it could be subclassed to override this behavior to take an already opened mmap object. This may satisfy my needs. I'm going to look into it and get back to you. In general, if you have a buffer() object, you can make an array from it using numpy.frombuffer(). This will be a standard ndarray and won't have the conveniences of syncing to disk that the memmap class provides. This is good to know because there have been a few situations when this would have been very useful. Suppose I do something like (in Python): import ctypes mylib = ctypes.CDLL('libmylib.so') y = mylib.get_float_array_from_c_function() which returns a float* as a Python int, and then I do nelems = mylib.get_float_array_num_elems() x = numpy.frombuffer(ctypes.c_buffer(y), 'float', nelems) This gives me an ndarray x with its (.data) buffer pointing to the memory address give by y. When the ndarray x is no longer referenced (even as another array's base), does numpy attempt to free the memory pointed to by y? In other words, does numpy always deallocate the (.data) buffer in the __del__ method? Or, does fromarray set a flag telling it not to? By default, frombuffer() creates an array that is flagged as not owning the data. That means it will not delete the data memory when the ndarray object is destroyed. In [69]: import ctypes In [70]: ca = (ctypes.c_int*8)() In [71]: a = frombuffer(ci, int) In [72]: a Out[72]: array([0, 0, 0, 0, 0, 0, 0, 0]) In [73]: a.flags Out[73]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False If you don't have a buffer() object, but just have a pointer allocated from some C code, then you *could* fake an object which exposes the __array_interface__() method to describe the memory. The numpy.asarray() constructor will use that to make an ndarray object that uses the specified memory. This is advanced stuff and difficult to get right because of memory ownership and object lifetime issues. Allocating memory in C code would be very useful for me. If I were to use such a numpy.asarray() function (seems the frombuffer you mentioned would also work as described above), Yes, if you can create the buffer object or something that obeys the buffer protocol. ctypes arrays work fine; ctypes pointers don't. it makes sense for the C code to be responsible for deallocating the memory, not numpy. I understand that I would need to ensure that the deallocation happens only when the containing ndarray is no longer referenced anywhere in Python (hopefully, ndarray's finalization code does not need access to the .data buffer). My experience has been that this is fairly difficult to do. If you have *complete* control of the ndarray object over its entire lifetime, then this is reasonable. If you don't, then you are going to run into (nondeterministic!) segfaulting bugs eventually. For example, if you are only using it as a temporary inside a function and never return it, this is fine. You will also need to be very careful about constructing views from the ndarray; these will need to be controlled, too. You will have a bug if you delete myarray but return reversed_array=myarray[::-1], for example. I see that you are using ctypes. Be sure to take a look at the .ctypes attribute on ndarrays. This allows you to get a ctypes pointer object from an array. This might help you use numpy to allocate
Re: [Numpy-discussion] Bug in numpy all() function
Dan Goodman wrote: Hi all, I think this is a bug (I'm running Numpy 1.0.3.1): from numpy import * def f(x): return False all(f(x) for x in range(10)) True I guess the all function doesn't know about generators? Yup. It works on arrays and things it can turn into arrays by calling the C API equivalent of numpy.asarray(). There's a ton of magic and special cases in asarray() in order to interpret nested Python sequences as arrays. That magic works fairly well when we have sequences with known lengths; it fails utterly when given an arbitrary iterator of unknown length. So we punt. Unfortunately, what happens then is that asarray() sees an object that it can't interpret as a sequence to turn into a real array, so it makes a rank-0 array with the iterator object as the value. This evaluates to True. It's possible that asarray() should raise an exception for generators, but it would be a special case. We wouldn't be able to test for arbitrary iterables. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] random enhancement
Neal Becker wrote: One thing missing from random is a mechanism to share a single underlying rng with other code that is not part of numpy.random. For example, I have code that generates distributions that expect a mersenne twister (the shared, underlying rng) to be passed in as a constructor argument. numpy.random shares a single rng amongst it's own distributions, but I don't see any way to share with others. Are you talking about C or Python? In Python, just instantiate numpy.random.RandomState() and pass it around. All of the functions in numpy.random are just aliases to the methods on a global RandomState() instance. C is a problem because the module is implemented in Pyrex, and RandomState is an extension type. I've tried working on exposing the C API as a PyCObject like numpy does, but it is incredibly tedious and, furthermore, is unlikely to capture the higher-level methods like multivariate_normal(). I believe that Cython has a way to automatically expose the C API of a Pyrex/Cython module, but I haven't had the time to investigate it. For everything but the higher level methods like multivariate_normal(), we might be able to expose the pointer to the rk_state struct on the RandomState object as a PyCObject and punt on exposing the API. The C user can copy the randomkit.[ch] and distributions.[ch] files into their own code and operate on the rk_state pointer with those functions. We may be thwarted by symbol conflicts on some platforms, but I'm not sure. Contributions are, of course, welcome. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bug in numpy all() function
Anne Archibald wrote: On 06/02/2008, Robert Kern [EMAIL PROTECTED] wrote: I guess the all function doesn't know about generators? Yup. It works on arrays and things it can turn into arrays by calling the C API equivalent of numpy.asarray(). There's a ton of magic and special cases in asarray() in order to interpret nested Python sequences as arrays. That magic works fairly well when we have sequences with known lengths; it fails utterly when given an arbitrary iterator of unknown length. So we punt. Unfortunately, what happens then is that asarray() sees an object that it can't interpret as a sequence to turn into a real array, so it makes a rank-0 array with the iterator object as the value. This evaluates to True. It's possible that asarray() should raise an exception for generators, but it would be a special case. We wouldn't be able to test for arbitrary iterables. Would it be possible for asarray() to pull out the first element from the iterable, make an array out of it, then assume that all other values out of the iterable will have the same shape (raising an error, of course, when they aren't)? I guess this has high foot-shooting potential, but is it that much worse than numpy's shpe-guessing generally? I'm skeptical. Personally, it comes down to this: if you provide code that implements this safely and efficiently without making a confusing API, I'm more than happy to consider it for inclusion. But I'm not going to spend time trying to write the code. It would be handy to be able to use an iterable to fill an array, so that you'd never need to store the values in anything else first: a = N.array((sin(N.pi*x/n) for x in xrange(n))) If n is large enough that storage matters, a = N.sin(N.linspace(0, np.pi, n)) is always faster, more memory efficient, and more readable. Remember that the array will have to be dynamically resized as we go through the iterator. The memory movement is going to wipe out much of the benefit of having an iterator in the first place. For 1D arrays, remember that we have numpy.fromiter() already, so we can test this. In [39]: import numpy as np In [40]: from math import sin In [41]: n = 10 In [42]: %timeit np.fromiter((sin(np.pi*x/n) for x in xrange(n)), float) 10 loops, best of 3: 11.5 µs per loop In [43]: %timeit np.sin(np.linspace(0, np.pi, n)) 1 loops, best of 3: 26.1 µs per loop In [44]: n = 100 In [45]: %timeit np.fromiter((sin(np.pi*x/n) for x in xrange(n)), float) 1 loops, best of 3: 84 µs per loop In [46]: %timeit np.sin(np.linspace(0, np.pi, n)) 1 loops, best of 3: 32.3 µs per loop In [47]: n = 1000 In [48]: %timeit np.fromiter((sin(np.pi*x/n) for x in xrange(n)), float) 1000 loops, best of 3: 794 µs per loop In [49]: %timeit np.sin(np.linspace(0, np.pi, n)) 1 loops, best of 3: 91.8 µs per loop So, for n=10, the generator wins, but is n=10 really the case that you want to use a generator for? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy-discussion Digest, Vol 17, Issue 15
Matthew, please do not reply to the digests. Think of them as read-only. If you want to start a new thread, send your mail, with a descriptive Subject line, to numpy-discussion@scipy.org . If you want to reply to individual messages, please turn digest delivery *off* and receive and respond to messages normally. Thank you. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Setting contents of buffer for array object
On Feb 10, 2008 5:15 PM, Matthew Brett [EMAIL PROTECTED] wrote: Hi, I am sorry if I have missed something obvious, but is there any way in python of doing this: import numpy as np a = np.arange(10) b = np.arange(10)+1 a.data = b.data # raises error, but I hope you see what I mean ? Not really, no. Can you describe your use case in more detail? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Setting contents of buffer for array object
On Feb 10, 2008 6:48 PM, Matthew Brett [EMAIL PROTECTED] wrote: import numpy as np a = np.arange(10) b = np.arange(10)+1 a.data = b.data # raises error, but I hope you see what I mean ? Not really, no. Can you describe your use case in more detail? Yes - I am just writing the new median implementation. To allow future optimization, I would like to have the same signature as mean(): def median(a, axis=0, dtype=None, out=None) (axis=0 to change to axis=None default at some point). To do this, I need to copy the results of the median calculation in the routine into the array object given by 'out' - when passed. Ah, I see. You definitely do not want to reassign the .data buffer in this case. An out= parameter does not reassign the memory location that the array object points to. It should use the allocated memory that was already there. It shouldn't copy anything at all; otherwise, median(x, out=out) is no better than out[:] = median(x). Personally, I don't think that a function should expose an out= parameter unless if it can make good on that promise of memory efficency. Can you show us the current implementation that you have? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] non-contiguous array error
On Feb 10, 2008 7:43 PM, Brad Malone [EMAIL PROTECTED] wrote: Hi, I am receiving a AttributeError: incompatible shape for a non-contiguous array error. A quick illustration of the type of code that gives me the error is shown below: from numpy import * list=[i for i in range(0,27)] c=array(list) c.shape=(3,3,3) d=fft.fftn(c) d.shape=(27) error here I suppose this has something to do with the fact that the fourier transform of c has imaginary parts, which affects the way the information is stored in memory, and this messed up the call to .shape? No. The problem is that (27) is not a tuple. Parentheses are also used for grouping expressions in Python, so a single-element tuple needs a comma to disambiguate. You want d.shape = (27,). -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Setting contents of buffer for array object
On Feb 10, 2008 7:17 PM, Matthew Brett [EMAIL PROTECTED] wrote: Ah, I see. You definitely do not want to reassign the .data buffer in this case. An out= parameter does not reassign the memory location that the array object points to. It should use the allocated memory that was already there. It shouldn't copy anything at all; otherwise, median(x, out=out) is no better than out[:] = median(x). Personally, I don't think that a function should expose an out= parameter unless if it can make good on that promise of memory efficency. I agree - but there are more efficient median algorithms out there which can make use of the memory efficiently. I wanted to establish the call signature to allow that. I don't feel strongly about it though. I say add the out= parameter when you use such an algorithm. But if you like, just use slice assignment for now. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New bug with setup,py develop
On Feb 11, 2008 3:49 AM, David Cournapeau [EMAIL PROTECTED] wrote: On Feb 11, 2008 5:38 PM, Robert Kern [EMAIL PROTECTED] wrote: On Feb 11, 2008 2:21 AM, Robert Kern [EMAIL PROTECTED] wrote: I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools develop command. In order to make sure that setuptools' develop works with numpy.distutils' build_src, we override the develop command to reinitialize the build_src command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice. Spoke too soon. It fails with r4772, too. Does it mean that it was already broken before numscons merge or not ? Yes. I think I figured out the problem. python setup.py develop works fine. However, I ran into the problem when I added develop to the end of a command line that already had build. So essentially, build_src gets run twice. I'm not sure there is anything that can be done about that given distutils' handling of options. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] CTypes: How to incorporate a library with shared library module?
On Feb 12, 2008 12:41 AM, Damian Eads [EMAIL PROTECTED] wrote: Robert Kern wrote: On Feb 12, 2008 12:14 AM, Damian Eads [EMAIL PROTECTED] wrote: David Cournapeau wrote: On Mon, 2008-02-11 at 22:50 -0700, Damian Eads wrote: Dear Lou, You may want to try using distutils or setuputils, which makes compiling extensions much easier. It does the hard work of finding out which flags are needed to compile extensions on the host platform. There are many examples on the web on how to use distutils to build C extensions (http://docs.python.org/ext/building.html). Unfortunately, this does not work. Distutils only knows how to build python extensions, not shared libraries. Depending on the platform, this is not the same thing, and mac os X is such a platform where both are not the same. cheers, David Really? distutils generates .so files for me, which I assume are shared libraries. FYI: I'm running Fedora 8 on an x86. Does distutils not generate a shared library on a mac? Python extension modules are shared libraries, yes. But they must follow a particular format, namely exposing a correct initmodulename function. distutils/setuptools only maked Python extension modules, not arbitrary shared libraries. Perhaps I was a bit too liberal in my use of the term extension module. Several small libraries for a project at work do not define the standard initmodulename function, and yet they build with distutils. I can load them into ctypes without any hitches. Perhaps distutils does not check for the presence of the initmodulename function and required data structures? I'll admit I may be abusing distutils by using it for something for which it wasn't designed. Yup. It usually works on Linux because the initmodulename bit isn't checked. On Windows, it is, and the build will fail. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] CTypes: How to incorporate a library with shared library module?
On Feb 12, 2008 12:14 AM, Damian Eads [EMAIL PROTECTED] wrote: David Cournapeau wrote: On Mon, 2008-02-11 at 22:50 -0700, Damian Eads wrote: Dear Lou, You may want to try using distutils or setuputils, which makes compiling extensions much easier. It does the hard work of finding out which flags are needed to compile extensions on the host platform. There are many examples on the web on how to use distutils to build C extensions (http://docs.python.org/ext/building.html). Unfortunately, this does not work. Distutils only knows how to build python extensions, not shared libraries. Depending on the platform, this is not the same thing, and mac os X is such a platform where both are not the same. cheers, David Really? distutils generates .so files for me, which I assume are shared libraries. FYI: I'm running Fedora 8 on an x86. Does distutils not generate a shared library on a mac? Python extension modules are shared libraries, yes. But they must follow a particular format, namely exposing a correct initmodulename function. distutils/setuptools only maked Python extension modules, not arbitrary shared libraries. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] CTypes: How to incorporate a library with shared library module?
On Feb 12, 2008 12:14 AM, Damian Eads [EMAIL PROTECTED] wrote: David Cournapeau wrote: On Mon, 2008-02-11 at 22:50 -0700, Damian Eads wrote: Dear Lou, You may want to try using distutils or setuputils, which makes compiling extensions much easier. It does the hard work of finding out which flags are needed to compile extensions on the host platform. There are many examples on the web on how to use distutils to build C extensions (http://docs.python.org/ext/building.html). Unfortunately, this does not work. Distutils only knows how to build python extensions, not shared libraries. Depending on the platform, this is not the same thing, and mac os X is such a platform where both are not the same. cheers, David Really? distutils generates .so files for me, which I assume are shared libraries. FYI: I'm running Fedora 8 on an x86. Does distutils not generate a shared library on a mac? As to David's point, yes, distutils makes a .so shared library on Macs. This is not the same thing as a dynamic library (on Macs) which is what ctypes needs (on Macs), IIRC. There is a subtle, but important difference between the two. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] New bug with setup,py develop
I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools develop command. In order to make sure that setuptools' develop works with numpy.distutils' build_src, we override the develop command to reinitialize the build_src command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice. This causes links to fail since the same symbol shows up twice. David, any ideas? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New bug with setup,py develop
On Feb 11, 2008 2:21 AM, Robert Kern [EMAIL PROTECTED] wrote: I've just updated the SVN trunk to get the latest numscons merge. Something broke the support I put in for the setuptools develop command. In order to make sure that setuptools' develop works with numpy.distutils' build_src, we override the develop command to reinitialize the build_src command to add the --inplace option. This used to work as of r4772, but now any Fortran Extensions have the generated sources added twice. Spoke too soon. It fails with r4772, too. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Median advice
On Feb 12, 2008 6:33 AM, Joris De Ridder [EMAIL PROTECTED] wrote: On 12 Feb 2008, at 12:31, Matthew Brett wrote: def median(a, axis=0, out=None) (same signature as max, min etc) I would be slightly in favour of this option. Using the same signature would be convenient in code like def myfunc(myarray, somefunc): # do stuff ... x = somefunc(myarray, axis = 0, out = None) # do more stuff ... where somefunc could be median(), mean(), max(), min(), std() etc. I once wrote this kind of function to provide (small) image filtering. If the same signature is used, there is no need to special-case median(). I realise it's kind of a niche example, though. I'm happy with that use case. The docstring should mention that out= is not memory-optimized like it is for the others, though. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unexpected downcast
Alan G Isaac wrote: On Tue, 12 Feb 2008, dmitrey apparently wrote: from numpy import * a = array((1.0, 2.0), float128) b=asfarray(a) type(a[0]) #type 'numpy.float128' type(b[0]) #type 'numpy.float64' __version__ '1.0.5.dev4767' Dmitrey noted an unexpected down cast (above). Is there a reason for it? That's just what asfarray is designed to do. If you don't give it a dtype, it uses float64. Changing it would be a redesign of the function that may break code. The amount of code is probably minimal, so I'm only -0 on changing it. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion