Re: [Numpy-discussion] Casting Bug or a Feature?
On 17.01.2013 04:43, Patrick Marsh wrote: Thanks, everyone for chiming in. Now that I know this behavior exists, I can explicitly prevent it in my code. However, it would be nice if a warning or something was generated to alert users about the inconsistency between var += ... and var = var + ... Patrick I agree wholeheartedly. I actually, for a long time, used to believe that python would translate a += b to a = a + b and was bitten several times by this bug. A warning (which can be silenced if you desperately want to) would be really nice, imho. Keep up the good work, Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Embedded NumPy LAPACK errors
On 4. jan. 2013, at 21:42, m...@eml.cc wrote: Hiall, I am trying to embed numerical code in a mexFunction, as called by MATLAB, written as a Cython function. NumPy core functions and BLAS work fine, but calls to LAPACK function such as SVD seem to be made against to MATLAB's linked MKL, and this generates MKL errors. When I try this with Octave, it works fine, presumably because it is compiled against the same LAPACK as the NumPy I am embedding. Assuming I haven't made big mistakes up to here, I have the following questions: Is there a way to request numpy.linalg to use a particular LAPACK library, e.g. /usr/lib/liblapack.so ? If not, is there a reasonable way to build numpy.linalg such that it interfaces with MKL correctly ? It's possible, but it's much easier to install one of the pre-built python distributions. Enthought, WinPython and others include precompiled python/numpy/scipy/etc with MKL. If that works for you, I'd recommend that route, as it involves less work. Good luck, Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.savez(_compressed) in loop
On 29. okt. 2012, at 11:29, Radek Machulka wrote:
Hi,
is there a way how to save more arrays into single npy (npz if possible) file
in loop? Something like:
fw = open('foo.bar', 'wb')
while foo:
arr = np.array(bar)
np.savez_compressed(fw, arr)
fw.close()
Or some workaround maybe? I go through hundreds of thousands arrays and can
not keep them in memory. Yes, I can save each array into single file, but I
would better have them all in single one.
As Pauli said, hdf5 is nicer for such things. I foresee:
import h5py
fw = h5py.File('foo.hdf5', 'w')
while foo:
arr = np.array(bar)
fw['arr'] = arr
fw.close()
Good luck
Paul
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: WinPython v2.7.3.0
Hi Pierre, first I'd like to congratulate you on a product which seems to finally solve all the problems I have at work regarding python scripting. A portable, fully featured python distribution is indeed very useful. One question though: Is there a way to run winpython's IPython under cygwin, without a separate window popping up? Just setting the PATH doesn't quite cut it, unfortunately. I'm interested because cygwin doesn't come with many science oriented python packages at all. Cheers Paul On 24. sep. 2012, at 21:22, Pierre Raybaut wrote: Hi all, I'm pleased to introduce my new contribution to the Python community: WinPython. WinPython v2.7.3.0 has been released and is available for 32-bit and 64-bit Windows platforms: http://code.google.com/p/winpython/ WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write use as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.x in the near future) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel). Enjoy! -Pierre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memory-efficient loadtxt
On 3. okt. 2012, at 17:48, Wes McKinney wrote: On Monday, October 1, 2012, Chris Barker wrote: Paul, Nice to see someone working on these issues, but: I'm not sure the problem you are trying to solve -- accumulating in a list is pretty efficient anyway -- not a whole lot overhead. But if you do want to improve that, it may be better to change the accumulating method, rather than doing the double-read thing. Ive written, and posted here, code that provides an Acumulator that uses numpy internally, so not much memory overhead. In the end, it's not any faster than accumulating in a list and then converting to an array, but it does use less memory. I also have a Cython version that is not quite done (darn regular job getting in the way) that is both faster and more memory efficient. Also, frankly, just writing the array pre-allocation and re-sizeing code into loadtxt would not be a whole lot of code either, and would be both fast and memory efficient. Let mw know if you want any of my code to play with. However, I got the impression that someone was working on a More Advanced (TM) C-based file reader, which will replace loadtxt; yes -- I wonder what happened with that? Anyone? -CHB this patch is intended as a useful thing to have while we're waiting for that to appear. The patch passes all tests in the test suite, and documentation for the kwarg has been added. I've modified all tests to include the seekable kwarg, but that was mostly to check that all tests are passed also with this kwarg. I guess it's bit too late for 1.7.0 though? Should I make a pull request? I'm happy to take any and all suggestions before I do. Cheers Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I've finally built a new, very fast C-based tokenizer/parser with type inference, NA-handling, etc. for pandas sporadically over the last month-- it's almost ready to ship. It's roughly an order of magnitude faster than loadtxt and uses very little temporary space. Should be easy to push upstream into NumPy to replace the innards of np.loadtxt if I can get a bit of help with the plumbing (it already yields structured arrays in addition to pandas DataFrames so there isn't a great deal that needs doing). Blog post with CPU and memory benchmarks to follow-- will post a link here. - Wes ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion So Chris, looks like Wes has us beaten in every conceivable way. Hey, that's a good thing :) I suppose the thing to do now is to make sure Wes' function tackles the loadtxt test suite? Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memory-efficient loadtxt
On 1. okt. 2012, at 21:07, Chris Barker wrote: Paul, Nice to see someone working on these issues, but: I'm not sure the problem you are trying to solve -- accumulating in a list is pretty efficient anyway -- not a whole lot overhead. Oh, there's significant overhead, since we're not talking of a list - we're talking of a list-of-lists. My guesstimate from my hacking session (off the top of my head - I don't have my benchmarks in working memory :) is around 3-5 times more memory with the list-of-lists approach for a single column / 1D array, which presumably is the worst case (a length 1 list for each line of input). Hence, if you want to load a 2 GB file into RAM on a machine with 4 GB of the stuff, you're screwed with the old approach and a happy camper with mine. But if you do want to improve that, it may be better to change the accumulating method, rather than doing the double-read thing. Ive written, and posted here, code that provides an Acumulator that uses numpy internally, so not much memory overhead. In the end, it's not any faster than accumulating in a list and then converting to an array, but it does use less memory. I see your point - but if you're to return a single array, and the file is close to the total system memory, you've still got a factor of 2 issue when shuffling the binary data from the accumulator into the result array. That is, unless I'm missong something here? Cheers Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memory-efficient loadtxt
On 3. okt. 2012, at 18:22, Chris Barker wrote: On Wed, Oct 3, 2012 at 9:05 AM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: I'm not sure the problem you are trying to solve -- accumulating in a list is pretty efficient anyway -- not a whole lot overhead. Oh, there's significant overhead, since we're not talking of a list - we're talking of a list-of-lists. hmm, a list of nupy scalers (custom dtype) would be a better option, though maybe not all that much better -- still an extra pointer and pyton object for each row. I see your point - but if you're to return a single array, and the file is close to the total system memory, you've still got a factor of 2 issue when shuffling the binary data from the accumulator into the result array. That is, unless I'm missong something here? Indeed, I think that's how my current accumulator works -- the __array__() method returns a copy of the data buffer, so that you won't accidentally re-allocate it under the hood later and screw up the returned version. But it is indeed accumulating in a numpy array, so it should be pretty possible, maybe even easy to turn it into a regular array without a data copy. You'd just have to destroy the original somehow (or mark it as never-resize) so you wouldn't have the clash. messing wwith the OWNDATA flags might take care of that. But it seems Wes has a better solution. Indeed. One other note, though -- if you have arrays that are that close to max system memory, you are very likely to have other trouble anyway -- numpy does make a lot of copies! That's true. Now, I'm not worried about this myself, but several people have complained about this on the mailing list, and it seemed like an easy fix. Oh well, it's too late for it now, anyways. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memory-efficient loadtxt
For convenience and clarity, this is the diff in question: https://github.com/Dynetrekk/numpy-1/commit/5bde67531a2005ef80a2690a75c65cebf97c9e00 And this is my numpy fork: https://github.com/Dynetrekk/numpy-1/ Paul On Sun, Sep 30, 2012 at 4:14 PM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: Hello everyone, I've modified loadtxt to make it (potentially) more memory efficient. The idea is that if a user passes a seekable file, (s)he can also pass the 'seekable=True' kwarg. Then, loadtxt will count the number of lines (containing data) and allocate an array of exactly the right size to hold the loaded data. The downside is that the line counting more than doubles the runtime, as it loops over the file twice, and there's a sort-of unnecessary np.array function call in the loop. The branch is called faster-loadtxt, which is silly due to the runtime doubling, but I'm hoping that the false advertising is acceptable :) (I naively expected a speedup by removing some needless list manipulation.) I'm pretty sure that the function can be micro-optimized quite a bit here and there, and in particular, the main for loop is a bit duplicated right now. However, I got the impression that someone was working on a More Advanced (TM) C-based file reader, which will replace loadtxt; this patch is intended as a useful thing to have while we're waiting for that to appear. The patch passes all tests in the test suite, and documentation for the kwarg has been added. I've modified all tests to include the seekable kwarg, but that was mostly to check that all tests are passed also with this kwarg. I guess it's bit too late for 1.7.0 though? Should I make a pull request? I'm happy to take any and all suggestions before I do. Cheers Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Second try: possible bug in assignment to complex array
On 10. aug. 2012, at 09:54, Mark Bakker wrote: I am giving this a second try. Can anybody help me out? I think there is a problem with assigning a 1D complex array of length one to a position in another complex array. Example: a = ones(1,'D') b = ones(1,'D') a[0] = b --- TypeError Traceback (most recent call last) ipython-input-37-0c4fc6d780e3 in module() 1 a[0] = b TypeError: can't convert complex to float This works correctly when a and b are real arrays: a = ones(1) b = ones(1) a[0] = b Bug or feature? The exact same thing happens on OS X 10.7.4, python 2.7.3, numpy 1.6.1. Looks like a bug to me - or at least very surprising behavior. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy Installation Problem on Redhat Linux
However, I got the following error message: error: Command /usr/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.6/build/src.linux-x86_64-2.6/scipy/integrate/vodemodule.o build/temp.linux-x86_64-2.6/build/src.linux-x86_64-2.6/fortranobject.o -L/home/username/lib/ -L/usr/lib64 -Lbuild/temp.linux-x86_64-2.6 -lodepack -llinpack_lite -lmach -lblas -lpython2.6 -lg2c -o build/lib.linux-x86_64-2.6/scipy/integrate/vode.so failed with exit status I'm sure there must have been more output? It does say that the command failed, but not _why_ it failed. I suggest posting the entire output either in an email, or on a webpage (gist.github.com, for instance) and giving the link. It's very very hard to debug a build without the build log, so I'd suggest always giving it in the first instance. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy Installation Problem on Redhat Linux
Hi, are you sure that you want g77 and not gfortran? If you want gfortran, you should pass the --fcompiler=gnu95 flag to setup.py. Which redhat version are you building on? (I don't know red hat well enough to comment, but perhaps someone else do...) Paul On 6. juli 2012, at 03:00, Hung-Hsuan Chen wrote: Dear all, I've built blas, lapack, and atlas libraries, as shown below. $ ls ~/lib/atlas/lib/ libatlas.a libcblas.a libf77blas.a liblapack.a libptcblas.a libptf77blas.a The library location are specified by site.cfg file, as shown below. [DEFAULT] library_dirs = /home/username/lib/atlas/lib include_dirs = /home/username/lib/atlas/include [blas] libraries = libf77blas, libcblas, libatlas [lapack] libraries = liblapack, libf77blas, libcblas, libatlas I've tried to build numpy (version 1.6.2) by $ python setup.py build --fcompiler=gnu However, I got the following error message: error: Command /usr/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.6/build/src.linux-x86_64-2.6/scipy/integrate/vodemodule.o build/temp.linux-x86_64-2.6/build/src.linux-x86_64-2.6/fortranobject.o -L/home/username/lib/ -L/usr/lib64 -Lbuild/temp.linux-x86_64-2.6 -lodepack -llinpack_lite -lmach -lblas -lpython2.6 -lg2c -o build/lib.linux-x86_64-2.6/scipy/integrate/vode.so failed with exit status I've searched internet for possible solutions whole day but don't have any progress so far. Anyone has any idea of how to fix this? Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy and readline installation fails
Hello, I don't know exactly what went wrong. I'd start out my debugging by 1) which python # see whether you're running apple's python in /usr/bin/python, or the one you tried to install 2) which easy_install # did you run Apple-python's easy_install, or the one you (tried to) installed? 3) If all of the above match, try running python (not ipython) and try to import numpy. Apple's python ships with numpy (at least the Lion / 10.7 one does). 4) Next, print numpy.__file__ to see whether numpy got installed to where it should In general, I'd advice you to install one package at a time, then test it to see whether it has been installed properly. When you're confident everything is OK, move on to the next package. For instance, test numpy by $ python -c 'import numpy; numpy.test()' and scipy with $ python -c 'import scipy;scipy.test()' (for instance). When you're sure the fundament (python, numpy) is in order, proceed with the house (scipy, matplotlib). Cheers Paul On 4. juli 2012, at 16:16, abc def wrote: Hello, I'm new to python and I'd like to learn about numpy / scipy / matplotlib, but I'm having trouble getting started. I'm following the instructions here: http://www.scipy.org/Getting_Started First I installed the latest version of python from python.org by downloading the dmg file, since I read that it doesn't work with apple's installer, and then installed numpy / scipy / matplotlib by downloading the relevent dmg files. I then downloaded ipython, ran easy_install readline and then ran python setup.py install. Then I started ipython with ipython -pylab as per the instructions but then I get muliple error messages: $ ipython --pylab /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/utils/rlineimpl.py:111: RuntimeWarning: ** libedit detected - readline will not be well behaved, including but not limited to: * crashes on tab completion * incorrect history navigation * corrupting long-lines * failure to wrap or indent lines properly It is highly recommended that you install readline, which is easy_installable: easy_install readline Note that `pip install readline` generally DOES NOT WORK, because it installs to site-packages, which come *after* lib-dynload in sys.path, where readline is located. It must be `easy_install readline`, or to a custom location on your PYTHONPATH (even --user comes after lib-dyload). ** RuntimeWarning) Python 2.7.3 (v2.7.3:70274d53c1dd, Apr 9 2012, 20:52:43) Type copyright, credits or license for more information. IPython 0.13 -- An enhanced Interactive Python. ? - Introduction and overview of IPython's features. %quickref - Quick reference. help - Python's own help system. object? - Details about 'object', use 'object??' for extra details. [TerminalIPythonApp] GUI event loop or pylab initialization failed --- ImportError Traceback (most recent call last) /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/core/pylabtools.pyc in find_gui_and_backend(gui) 194 195 -- 196 import matplotlib 197 198 if gui and gui != 'auto': /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/__init__.py in module() 131 import sys, os, tempfile 132 -- 133 from matplotlib.rcsetup import (defaultParams, 134 validate_backend, 135 validate_toolbar, /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/rcsetup.py in module() 17 import warnings 18 from matplotlib.fontconfig_pattern import parse_fontconfig_pattern --- 19 from matplotlib.colors import is_color_like 20 21 #interactive_bk = ['gtk', 'gtkagg', 'gtkcairo', 'fltkagg', 'qtagg', 'qt4agg', /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/colors.py in module() 50 51 import re --- 52 import numpy as np 53 from numpy import ma 54 import matplotlib.cbook as cbook ImportError: No module named numpy In [1]: it seems the installation of numpy and readline didn't work, and there are problems with matplotlib, even though I think I followed all the instructions carefully. I can't figure out what I did wrong. Can anybody help? I'm running mac os 10.6. Thank you! Tom ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] better error message possible?
On 7. juni 2012, at 10:30, Thouis (Ray) Jones wrote: I've opened a PR at https://github.com/numpy/numpy/pull/296 for discussion. A typical result np.zeros((3,3))[[1,2,3]] Traceback (most recent call last): File stdin, line 1, in module IndexError: index 3 is out of bounds for axis 0: [-3,3) Ray Jones ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I would prefer: IndexError: index 3 is out of bounds for axis 0: [-3,2] as I find the 3) notation a bit weird - after all, indices are not floats, so 2.999 or 2.3 doesn't make sense as an index. An alternative is to not refer to negative indices and say IndexError: index 3 is out of bounds for axis 0, shape was: (3,) (print more axes when the number of axes is higher.) BTW, I'm really glad someone is taking an interest in these error messages, it's a great idea! Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] better error message possible?
On 4. juni 2012, at 16:27, Thouis (Ray) Jones wrote: On Fri, Jun 1, 2012 at 6:56 PM, Chris Withers ch...@simplistix.co.uk wrote: On 01/06/2012 16:39, Benjamin Root wrote: import numpy numpy.zeros(10)[-123] Traceback (most recent call last): File stdin, line 1, in module IndexError: index out of bounds ...could say this: numpy.zeros(10)[-123] Traceback (most recent call last): File stdin, line 1, in module IndexError: -123 is out of bounds Only that no-one has implemented it, I guess. If you want to then that'd be cool :-). To be generally useful for debugging, it would probably be good for the error message to also mention which dimension is involved, and/or the actual size of the array in that dimension. You can also get such error messages from expressions like 'arr[i, j, k]', after all, where it's even less obvious what went wrong. -- Nathaniel +1, please! Indeed, sadly I'm not a C developer. It's a pet bugbear of mine that Python's built-in exceptions often tell you what went wrong but not what data caused the error, even when it's easily to hand when raising the exception. I could look into this. There are only ~10 places the code generates this error, so it should be a pretty minor change. Ray Jones Isn't it useful even if you change it in just one of those locations? Better to have the information available when you can, than to never have it. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: NumPy 1.6.2 release candidate 1
On Mon, May 14, 2012 at 9:47 PM, Ralf Gommers
ralf.gomm...@googlemail.com wrote:
On Sun, May 13, 2012 at 1:14 PM, Paul Anton Letnes
paul.anton.let...@gmail.com wrote:
On Sat, May 12, 2012 at 9:50 PM, Ralf Gommers
ralf.gomm...@googlemail.com wrote:
On Sun, May 6, 2012 at 12:12 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Sat, May 5, 2012 at 2:56 PM, Paul Anton Letnes
paul.anton.let...@gmail.com wrote:
Hi,
I'm getting a couple of errors when testing. System:
Arch Linux (updated today)
Python 3.2.3
gcc 4.7.0
(Anything else?)
I think that this error:
AssertionError: selectedrealkind(19): expected -1 but got 16
is due to the fact that newer versions of gfortran actually supports
precision this high (quad precision).
Yes, but it should be fixed. I can't duplicate this here with a fresh
checkout of the branch.
This failure makes no sense to me.
Error comes from this code:
'selectedrealkind(%s): expected %r but got %r' % (i,
selected_real_kind(i), selectedrealkind(i)))
So selected_real_kind(19) returns -1.
selected_real_kind is the function
numpy.f2py.crackfortran._selected_real_kind_func, which is defined as:
def _selected_real_kind_func(p, r=0, radix=0):
#XXX: This should be processor dependent
# This is only good for 0 = p = 20
if p 7: return 4
if p 16: return 8
if platform.machine().lower().startswith('power'):
if p = 20:
return 16
else:
if p 19:
return 10
elif p = 20:
return 16
return -1
For p=19 this function should always return 16. So the result from
compiling
foo.f90 is fine, but the test is broken in a very strange way.
Paul, is the failure reproducible on your machine? If so, can you try to
debug it?
Ralf
Hi Ralf.
The Arch numpy (1.6.1) for python 2.7, installed via pacman (the
package manager) has this problem.
After installation of numpy 1.6.2rc1 in a virtualenv, the test passes.
Maybe the bug was fixed in the RC, and I screwed up which numpy
version I tested? I'm sorry that I can't find out - I just built a new
machine, and the old one is lying around the livingroom in pieces. Was
that particular bit of code changed between 1.6.1 and 1.6.2rc1?
It was actually, in https://github.com/numpy/numpy/commit/e7f2210e1.
So you tested 1.6.1 by accident before, and it's working now? Problem solved
in that case.
Ralf
Looks like it to me! Sorry for the silly bug report. I'll have to take
more care next time...
Paul
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: NumPy 1.6.2 release candidate 1
On Sat, May 12, 2012 at 9:50 PM, Ralf Gommers
ralf.gomm...@googlemail.com wrote:
On Sun, May 6, 2012 at 12:12 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Sat, May 5, 2012 at 2:56 PM, Paul Anton Letnes
paul.anton.let...@gmail.com wrote:
Hi,
I'm getting a couple of errors when testing. System:
Arch Linux (updated today)
Python 3.2.3
gcc 4.7.0
(Anything else?)
I think that this error:
AssertionError: selectedrealkind(19): expected -1 but got 16
is due to the fact that newer versions of gfortran actually supports
precision this high (quad precision).
Yes, but it should be fixed. I can't duplicate this here with a fresh
checkout of the branch.
This failure makes no sense to me.
Error comes from this code:
'selectedrealkind(%s): expected %r but got %r' % (i,
selected_real_kind(i), selectedrealkind(i)))
So selected_real_kind(19) returns -1.
selected_real_kind is the function
numpy.f2py.crackfortran._selected_real_kind_func, which is defined as:
def _selected_real_kind_func(p, r=0, radix=0):
#XXX: This should be processor dependent
# This is only good for 0 = p = 20
if p 7: return 4
if p 16: return 8
if platform.machine().lower().startswith('power'):
if p = 20:
return 16
else:
if p 19:
return 10
elif p = 20:
return 16
return -1
For p=19 this function should always return 16. So the result from compiling
foo.f90 is fine, but the test is broken in a very strange way.
Paul, is the failure reproducible on your machine? If so, can you try to
debug it?
Ralf
Hi Ralf.
The Arch numpy (1.6.1) for python 2.7, installed via pacman (the
package manager) has this problem.
After installation of numpy 1.6.2rc1 in a virtualenv, the test passes.
Maybe the bug was fixed in the RC, and I screwed up which numpy
version I tested? I'm sorry that I can't find out - I just built a new
machine, and the old one is lying around the livingroom in pieces. Was
that particular bit of code changed between 1.6.1 and 1.6.2rc1?
Paul
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: NumPy 1.6.2 release candidate 1
All tests for 1.6.2rc1 pass on Mac OS X 10.7.3 python 2.7.2 gcc 4.2 (Apple) Great! Paul On 6. mai 2012, at 00:12, Charles R Harris wrote: On Sat, May 5, 2012 at 2:56 PM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: Hi, I'm getting a couple of errors when testing. System: Arch Linux (updated today) Python 3.2.3 gcc 4.7.0 (Anything else?) I think that this error: AssertionError: selectedrealkind(19): expected -1 but got 16 is due to the fact that newer versions of gfortran actually supports precision this high (quad precision). Yes, but it should be fixed. I can't duplicate this here with a fresh checkout of the branch. snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: NumPy 1.6.2 release candidate 1
Hi, I'm getting a couple of errors when testing. System: Arch Linux (updated today) Python 3.2.3 gcc 4.7.0 (Anything else?) I think that this error: AssertionError: selectedrealkind(19): expected -1 but got 16 is due to the fact that newer versions of gfortran actually supports precision this high (quad precision). Cheers Paul python -c 'import numpy;numpy.test(full)' Running unit tests for numpy NumPy version 1.6.1 NumPy is installed in /usr/lib/python3.2/site-packages/numpy Python version 3.2.3 (default, Apr 23 2012, 23:35:30) [GCC 4.7.0 20120414 (prerelease)] nose version 1.1.2 S.S..SSS...KK.K.K..K..F../usr/lib/python3.2/site-packages/numpy/lib/format.py:575: ResourceWarning: unclosed file _io.BufferedReader name='/tmp/tmpmkxhkq' mode=mode, offset=offset) /usr/lib/python3.2/subprocess.py:471: ResourceWarning: unclosed file _io.FileIO name=3 mode='rb' return Popen(*popenargs, **kwargs).wait() /usr/lib/python3.2/subprocess.py:471: ResourceWarning: unclosed file _io.FileIO name=7 mode='rb' return Popen(*popenargs, **kwargs).wait() ..F... == FAIL: test_kind.TestKind.test_all -- Traceback (most recent call last): File /usr/lib/python3.2/site-packages/nose/case.py, line 198, in runTest self.test(*self.arg) File /usr/lib/python3.2/site-packages/numpy/f2py/tests/test_kind.py, line 30, in test_all
Re: [Numpy-discussion] timing results (was: record arrays initialization)
On Fri, May 4, 2012 at 12:49 AM, Keith Goodman kwgood...@gmail.com wrote: On Thu, May 3, 2012 at 3:12 PM, Moroney, Catherine M (388D) catherine.m.moro...@jpl.nasa.gov wrote: Here is the python code: def single(element, targets): if (isinstance(element, tuple)): xelement = element[0] elif (isinstance(element, numpy.ndarray)): xelement = element else: return FILL nlen = xelement.shape[0] nvec = targets.data.shape[0] x = xelement.reshape(1, nlen).repeat(nvec, axis=0) repeat is slow. I don't think you need it since broadcasting should take care of things. (But maybe I misunderstand the data.) diffs = ((x - targets.data)**2).sum(axis=1) You could try np.dot instead of sum: one = np.ones(7); diff = np.dot(diff, one). You could even pass one in. diffs = numpy.sqrt(diffs) Since you don't return the distance, no need for the sqrt. If you do need the sqrt only take the sqrt of one element instead of all elements. return int(numpy.argmin(diffs, axis=0)) As Keith says, you don't have to do the sqrt of everything. I'd probably set the min_dist to be a negative value initially. Perhaps you'll encounter large distances at some point, and negative values are more obviously incorrect when you're debugging. If you're 100% sure you're setting every single line of the array matches, then you can skip this line: matches(:) = - It won't buy you much, but if it's for free... Two suggestions. Either: dvector = targets(:,it) - vectors(:,iv) dist = sqrt(sum(dvector*dvector)) If you rewrite it in a loop as dist = 0 do itemp = 1, size(targets, 1) dist = dist + (targets(itemp, it) - vectors(itemp, iv))**2 end do (I am skipping the sqrt as it is not needed.) That way you're not looping over the array three times. You will be doing the sum, the difference, and the exponentiation all in one loop. Moving memory around is typically more expensive than computing. A good Fortran compiler should optimize **2 into multiplication by itself instead of calling a library routine, too. Alternatively: dist = sqrt(sum(dvector*dvector)) Can be written as (skipping the sqrt again): dist = dot_product(dvector, dvector) or: dist = DDOT(size(dvector), dvector, 1, dvector, 1) In fortran there's the builtin dot_product function. If this is in fact one of your bottlenecks, you should also try calling DDOT (= dot product) from an optimized BLAS library. This is what numpy.dot does, too, but in Fortran you avoid calling overhead. Hope this helps - it's kind of hard to know exactly what works without playing with the code yourself :) Also, sometimes your compiler has some of these things already and you don't get anything by hand-optimizing. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] timing results (was: record arrays initialization)
On 3. mai 2012, at 19:33, Moroney, Catherine M (388D) wrote: A quick recap of the problem: a 128x512 array of 7-element vectors (element), and a 5000-vector training dataset (targets). For each vector in element, I want to find the best-match in targets, defined as minimizing the Euclidean distance. I coded it up three ways: (a) looping through each vector in element individually, (b) vectorizing the function in the previous step, and coding it up in Fortran. The heart of the find-best-match code in Python looks like so I'm not doing an individual loop through all 5000 vectors in targets: nlen = xelement.shape[0] nvec = targets.data.shape[0] x = xelement.reshape(1, nlen).repeat(nvec, axis=0) diffs = ((x - targets.data)**2).sum(axis=1) diffs = numpy.sqrt(diffs) return int(numpy.argmin(diffs, axis=0)) Here are the results: (a) looping through each vector: 68 seconds (b) vectorizing this: 58 seconds (c) raw Fortran with loops: 26 seconds I was surprised to see that vectorizing didn't gain me that much time, and that the Fortran was so much faster than both python alternatives. So, there's a lot that I don't know about how the internals of numpy and python work. Why does the loop through 128x512 elements in python only take an additional 10 seconds? What is the main purpose of vectorizing - is it optimization by taking the looping step out of the Python and into the C-base or something different? If you're doing loops in python, python does all sort of magic for you. Type checking is one thing, and one of the simplest things to optimize away if you use cython. If you're writing an expression similar to this ((x - targets.data)**2) where x and targets.data are vectors, the elements are subtracted and squared elementwise in C instead of in python. So yes, you've got the idea. And, why is the Fortran so much faster (even without optimization)? Could you show us the code? It's hard to tell otherwise. As Keith Goodman pointed out, if he gets 7.5x with cython, it could be that the Fortran code could be improved as well. Fortran has a reputation of being the gold standard for performance in numerical computation, although one can often be surprised. Picking good algorithms is always more important than the language. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
On 21. apr. 2012, at 00:16, Drew Frank wrote: On Fri, Apr 20, 2012 at 11:45 AM, Chris Barker chris.bar...@noaa.gov wrote: On Fri, Apr 20, 2012 at 11:39 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: Oh, right. I was thinking small as in fits in L2 cache, not small as in a few dozen entries. Another example of a small array use-case: I've been using numpy for my research in multi-target tracking, which involves something like a bunch of entangled hidden markov models. I represent target states with small 2d arrays (e.g. 2x2, 4x4, ..) and observations with small 1d arrays (1 or 2 elements). It may be possible to combine a bunch of these small arrays into a single larger array and use indexing to extract views, but it is much cleaner and more intuitive to use separate, small arrays. It's also convenient to use numpy arrays rather than a custom class because I use the linear algebra functionality as well as integration with other libraries (e.g. matplotlib). I also work with approximate probabilistic inference in graphical models (belief propagation, etc), which is another area where it can be nice to work with many small arrays. In any case, I just wanted to chime in with my small bit of evidence for people wanting to use numpy for work with small arrays, even if they are currently pretty slow. If there were a special version of a numpy array that would be faster for cases like this, I would definitely make use of it. Drew Although performance hasn't been a killer for me, I've been using numpy arrays (or matrices) for Mueller matrices [0] and Stokes vectors [1]. These describe the polarization of light and are always 4x1 vectors or 4x4 matrices. It would be nice if my code ran in 1 night instead of one week, although this is still tolerable in my case. Again, just an example of how small-vector/matrix performance can be important in certain use cases. Paul [0] https://en.wikipedia.org/wiki/Mueller_calculus [1] https://en.wikipedia.org/wiki/Stokes_vector ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py with int8
Hi, 1, 2, 3 are integer literals. 1.0, 3.0e2, -42.0 are real (float) literals 'hello world' is a string literal. As far as I remember, f2py requires a literal variable for the kind. The solution I have landed on is to write a pure fortran module (using int8, or whatever), and then wrap this module either with an f2py compatible fortran module or an interface file. If you want to know what int8 corresponds to, run the following (pure fortran) program through your compiler of choice: program kind_values use iso_fortran_env implicit none print *, 'int8 kind value:', int8 print *, 'int16 kind value:', int16 end program kind_values Paul On 17. apr. 2012, at 19:47, John Mitchell wrote: Thanks Paul. I suppose this is now going slightly out of bounds for f2py. What I am looking for is the fortran kind type for a byte. I thought that this was int8. I guess the question is how to identify the kind type. Although I have verified that integer(1) seems to work for me, I would really like to know why and your answer alludes to that. Please excuse my ignorance on this topic. Can you perhaps educate me a little on 'literal kind values'? I take you to mean that 'int8' is not a literal kind value while 1 and 8 are examples of literal kind values. Thanks, John On Tue, Apr 17, 2012 at 10:12 AM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: Ah, come to think of it, I think that f2py only supports literal kind values. Maybe that's your problem. Paul On 17. apr. 2012, at 07:58, Sameer Grover wrote: On Tuesday 17 April 2012 11:02 AM, John Mitchell wrote: Hi, I am using f2py to pass a numpy array of type numpy.int8 to fortran. It seems like I am misunderstanding something because I just can't make it work. Here is what I am doing. PYTHON b=numpy.array(numpy.zeros(shape=(10,),dtype=numpy.int8),order='F') b[0]=1 b[2]=1 b[3]=1 b array([1, 0, 1, 1, 0, 0, 0, 0, 0, 0], dtype=int8) FORTRAN subroutine print_bit_array(bits,n) use iso_fortran_env integer,intent(in)::n integer(kind=int8),intent(in),dimension(n)::bits print*,'bits = ',bits end subroutine print_bit_array RESULT when calling fortran from python bits = 1000000010 Any Ideas? thanks, John ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion It seems to work if integer(kind=int8) is replaced with integer(8) or integer(1). Don't know why, though. Sameer ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py with int8
Ah, come to think of it, I think that f2py only supports literal kind values. Maybe that's your problem. Paul On 17. apr. 2012, at 07:58, Sameer Grover wrote: On Tuesday 17 April 2012 11:02 AM, John Mitchell wrote: Hi, I am using f2py to pass a numpy array of type numpy.int8 to fortran. It seems like I am misunderstanding something because I just can't make it work. Here is what I am doing. PYTHON b=numpy.array(numpy.zeros(shape=(10,),dtype=numpy.int8),order='F') b[0]=1 b[2]=1 b[3]=1 b array([1, 0, 1, 1, 0, 0, 0, 0, 0, 0], dtype=int8) FORTRAN subroutine print_bit_array(bits,n) use iso_fortran_env integer,intent(in)::n integer(kind=int8),intent(in),dimension(n)::bits print*,'bits = ',bits end subroutine print_bit_array RESULT when calling fortran from python bits = 1000000010 Any Ideas? thanks, John ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion It seems to work if integer(kind=int8) is replaced with integer(8) or integer(1). Don't know why, though. Sameer ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py with int8
Hi, this probably does not help with your problem. However, I would recommend changing your fortran code to: subroutine print_bit_array(bits) use iso_fortran_env integer(kind=int8),intent(in),dimension(:)::bits print*,'bits = ',bits end subroutine print_bit_array In that way you could print shape(bits) to verify that you are getting an array of the size you are expecting. Also, you could compile with -fbounds-check (gfortran) or a similar flag for some extra debugging facilities. To get better help with your issues, I would recommend also posting your call to the fortran routine, and the compilation command used (f2py -m myfile.f90 -flags). Cheers Paul On 17. apr. 2012, at 07:32, John Mitchell wrote: Hi, I am using f2py to pass a numpy array of type numpy.int8 to fortran. It seems like I am misunderstanding something because I just can't make it work. Here is what I am doing. PYTHON b=numpy.array(numpy.zeros(shape=(10,),dtype=numpy.int8),order='F') b[0]=1 b[2]=1 b[3]=1 b array([1, 0, 1, 1, 0, 0, 0, 0, 0, 0], dtype=int8) FORTRAN subroutine print_bit_array(bits,n) use iso_fortran_env integer,intent(in)::n integer(kind=int8),intent(in),dimension(n)::bits print*,'bits = ',bits end subroutine print_bit_array RESULT when calling fortran from python bits = 1000000010 Any Ideas? thanks, John ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
As others on this list, I've also been confused a bit by the prolific numpy
interfaces to reading text. Would it be an idea to create some sort of object
oriented solution for this purpose?
reader = np.FileReader('my_file.txt')
reader.loadtxt() # for backwards compat.; np.loadtxt could instantiate a reader
and call this function if one wants to keep the interface
reader.very_general_and_typically_slow_reading(missing_data=True)
reader.my_files_look_like_this_plz_be_fast(fmt='%20.8e', separator=',', ncol=2)
reader.cvs_read() # same as above, but with sensible defaults
reader.lazy_read() # returns a generator/iterator, so you can slice out a small
part of a huge array, for instance, even when working with text (yes,
inefficient)
reader.convert_line_by_line(myfunc) # line-by-line call myfunc, letting the
user somehow convert easily to his/her format of choice: netcdf, hdf5, ... Not
fast, but convenient
Another option is to create a hierarchy of readers implemented as classes. Not
sure if the benefits outweigh the disadvantages.
Just a crazy idea - it would at least gather all the file reading interfaces
into one place (or one object hierarchy) so folks know where to look. The whole
numpy namespace is a bit cluttered, imho, and for newbies it would be
beneficial to use submodules to a greater extent than today - but that's a more
long-term discussion.
Paul
On 23. feb. 2012, at 21:08, Travis Oliphant wrote:
This is actually on my short-list as well --- it just didn't make it to the
list.
In fact, we have someone starting work on it this week. It is his first
project so it will take him a little time to get up to speed on it, but he
will contact Wes and work with him and report progress to this list.
Integration with np.loadtxt is a high-priority. I think loadtxt is now the
3rd or 4th text-reading interface I've seen in NumPy. I have no interest
in making a new one if we can avoid it. But, we do need to make it faster
with less memory overhead for simple cases like Wes describes.
-Travis
On Feb 23, 2012, at 1:53 PM, Pauli Virtanen wrote:
Hi,
23.02.2012 20:32, Wes McKinney kirjoitti:
[clip]
To be clear: I'm going to do this eventually whether or not it
happens in NumPy because it's an existing problem for heavy
pandas users. I see no reason why the code can't emit structured
arrays, too, so we might as well have a common library component
that I can use in pandas and specialize to the DataFrame internal
structure.
If you do this, one useful aim could be to design the code such that it
can be used in loadtxt, at least as a fast path for common cases. I'd
really like to avoid increasing the number of APIs for text file loading.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On 20. feb. 2012, at 16:29, Sturla Molden wrote: Den 20.02.2012 08:35, skrev Paul Anton Letnes: In the language wars, I have one question. Why is Fortran not being considered? Fortran already implements many of the features that we want in NumPy: Yes ... but it does not make Fortran a systems programming language. Making NumPy is different from using it. - slicing and similar operations, at least some of the fancy indexing kind - element-wise array operations and function calls - array bounds-checking and other debugging aid (with debugging flags) That is nice for numerical computing, but not really needed to make NumPy. - arrays that mentally map very well onto numpy arrays. To me, this spells +1 to ease of contribution, over some abstract C/C++ template Mentally perhaps, but not binary. NumPy needs uniformly strided memory on the binary level. Fortran just gives this at the mental level. E.g. there is nothing that dictates a Fortran pointer has to be a view, the compiler is free to employ copy-in copy-out. In Fortran, a function call can invalidate a pointer. One would therefore have to store the array in an array of integer*1, and use the intrinsic function transfer() to parse the contents into NumPy dtypes. - in newer standards it has some nontrivial mathematical functions: gamma, bessel, etc. that numpy lacks right now That belongs to SciPy. I don't see exactly why. Why should numpy have exponential but not gamma functions? The division seems kinda arbitrary. Not that I am arguing violently for bessel functions in numpy. - compilers that are good at optimizing for floating-point performance, because that's what Fortran is all about Insanely good, but not when we start to do the (binary, not mentally) strided access that NumPy needs. (Not that C compilers would be any better.) - not Fortran as such, but BLAS and LAPACK are easily accessed by Fortran - possibly other numerical libraries that can be helpful - Fortran has, in its newer standards, thought of C interoperability. We could still keep bits of the code in C (or even C++?) if we'd like to, or perhaps f2py/Cython could do the wrapping. Not f2py, as it depends on NumPy. - some programmers know Fortran better than C++. Fortran is at least used by many science guys, like me. That is a valid arguments. Fortran is also much easier to read and debug. Sturla Thanks for an excellent answer, Sturla - very informative indeed. Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
In the language wars, I have one question. Why is Fortran not being considered? Fortran already implements many of the features that we want in NumPy: - slicing and similar operations, at least some of the fancy indexing kind - element-wise array operations and function calls - array bounds-checking and other debugging aid (with debugging flags) - arrays that mentally map very well onto numpy arrays. To me, this spells +1 to ease of contribution, over some abstract C/C++ template - in newer standards it has some nontrivial mathematical functions: gamma, bessel, etc. that numpy lacks right now - compilers that are good at optimizing for floating-point performance, because that's what Fortran is all about - not Fortran as such, but BLAS and LAPACK are easily accessed by Fortran - possibly other numerical libraries that can be helpful - Fortran has, in its newer standards, thought of C interoperability. We could still keep bits of the code in C (or even C++?) if we'd like to, or perhaps f2py/Cython could do the wrapping. - some programmers know Fortran better than C++. Fortran is at least used by many science guys, like me. Until someone comes along with actual numbers or at least anecdotal evidence, I don't think the more programmers know X than Y argument is too interesting. Personally I've learned both, and Fortran is much more accessible than C++ (to me) if you're used to the work with (numpy) arrays mentality. As far as I can understand, implementing element-wise operations, slicing, and a host of other NumPy features is in some sense pointless - the Fortran compiler authors have already done it for us. Of course some nice wrapping will be needed in C, Cython, f2py, or similar. Since my understanding is limited, I'd be interested in being proved wrong, though :) Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy governance update
An example I really like is LibreOffice's get involved page. http://www.libreoffice.org/get-involved/ Producing something similar for NumPy will take some work, but I believe it's needed. Speaking as someone who has contributed to numpy in a microscopic fashion, I agree completely. I spent quite a few hours digging through the webpages, asking for help on the mailing list, reading the Trac, reading git tutorials etc. before I managed to do something remotely useful. In general, I think the webpage for numpy (and scipy, but let's not discuss that here) would benefit from some refurbishing, including the documentation pages. As an example, one of the links on the webpage is Numpy for MATLAB users. I never used matlab much, so this is completely irrelevant for me. I think there should be a discussion about what goes on the front page, and it should be as little as possible, but not less than that. Make it easy for people to 1) start using numpy 2) reading detailed documentation 3) reporting bugs 4) contributing to numpy because those are the fundamental things a user/developer wants from an open source project. Right now there's Trac, github, numpy.scipy.org, http://docs.scipy.org/doc/, the mailing list, and someone mentioned a google group discussing something or other. It took me years to figure out how things are patched together, and I'm still not sure exactly who reads the Trac discussion, github discussion, and mailing list discussions. tl;dr: Numpy is awesome (TM) but needs a more coherent online presence, and one that makes it easy to contribute back to the project. Thanks for making numpy awesome! Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to cite 1Xn array as nX1 array?
On 27. jan. 2012, at 14:52, Chao YUE wrote: Dear all, suppose I have a ndarray a: In [66]: a Out[66]: array([0, 1, 2, 3, 4]) how can use it as 5X1 array without doing a=a.reshape(5,1)? Several ways, this is one, although not much simpler. In [6]: a Out[6]: array([0, 1, 2, 3, 4]) In [7]: a.shape = 5, 1 In [8]: a Out[8]: array([[0], [1], [2], [3], [4]]) Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] array metadata
If by store you mean store on disk, I recommend h5py datasets and attributes. Reportedly pytables is also good but I don't have any first hand experience there. Both python modules use the hdf5 library, written in C/C++/Fortran. Paul On Wed, Jan 25, 2012 at 7:47 PM, Val Kalatsky kalat...@gmail.com wrote: I believe there are no provisions made for that in ndarray. But you can subclass ndarray. Val On Wed, Jan 25, 2012 at 12:10 PM, Emmanuel Mayssat emays...@gmail.com wrote: Is there a way to store metadata for an array? For example, date the samples were collected, name of the operator, etc. Regards, -- Emmanuel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.zeros(2, 'S') returns empty strings.
On 15. jan. 2012, at 01:21, josef.p...@gmail.com wrote:
On Sat, Jan 14, 2012 at 5:25 PM, Benjamin Root ben.r...@ou.edu wrote:
On Sat, Jan 14, 2012 at 4:16 PM, Benjamin Root ben.r...@ou.edu wrote:
On Sat, Jan 14, 2012 at 4:12 PM, Charles R Harris
charlesr.har...@gmail.com wrote:
This sort of makes sense, but is it the 'correct' behavior?
In [20]: zeros(2, 'S')
Out[20]:
array(['', ''],
dtype='|S1')
It might be more consistent to return '0' instead, as in
In [3]: zeros(2, int).astype('S')
Out[3]:
array(['0', '0'],
dtype='|S24')
I would be surprised if zeros is not an empty string, since an empty
string is the zero for string addition.
multiplication for strings doesn't exist, so ones can be anything even
literally '1'
My python disagrees.
In [1]: 2 * 'spam ham '
Out[1]: 'spam ham spam ham '
Not sure what the element-wise numpy array equivalent would be, though.
Paul
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What does fftn take as parameters?
On 6. des. 2011, at 17:32, Roger Binns wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/12/11 01:58, Pauli Virtanen wrote: I think this cannot be helped --- it does not make sense to explain basic Numpy concepts in every docstring, especially `axis` and `shape` are very common. They don't need to be explained on the page, but instead link to a page that does explain them. The test is that an experienced Python programmer should be able to understand what is going on from the fft doc page and every page it links to. Note that searching doesn't help: First google hit, numpy axis: http://www.scipy.org/Tentative_NumPy_Tutorial Under Basics, third and forth sentences: NumPy's main object is the homogeneous multidimensional array. It is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers. In Numpy dimensions are called axes. The number of axes is rank. Searching is useful indeed, just use google instead. You can do the several column FFT easily using the axes keyword. Have a look at that link. If you are using numpy functions like FFT, you are indeed converting to arrays. Python lists are poorly performing for numerical work and do not map well onto the C and Fortran based codes that give numpy and scipy their good performance. Hence, numpy will convert to arrays for you, and if you are treating the returned array as a list, you are probably (among other things) indexing it in an inefficient way. It would seem to me that if your obsession over the FFT must be misplaced. If you are passing multidimensional lists here and there, you must have a lot of overhead memory use. Since the FFT is so efficient, O(n log (n)), I'm guessing that it's not your bottleneck (as others have suggested). Try using the cProfile module (python stdlib) to profile your code, and verify where your bottlenecks are. Come back to this list for more advice when you have evidence for where your bottleneck actually is. As a side note: since the built-in search isn't really all that good, would it be possible to put a customized google search box there instead? Good luck Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading automatically all the parameters from a file
On 30. nov. 2011, at 12:09, Giovanni Plantageneto wrote: Dear all, I have a simple question. I would like to have all the parameters of a model written in a configuration file (text), and I would like to have all the parameters in the file automatically defined inside a program. I find ConfigParser a bit low level, is there any function that automatically reads everything from a file? Thanks. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I like having my input files simply be python files, on the form bar = 'foo' ham = ['spam', 'eggs'] Then I import them as import imp parameters = imp.load_source(parameters, myinpufile.py) Now the object 'parameters' is a python module so I can say print parameters.bar and 'foo' will be printed. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype
Hi, I don't know about documentation, but I always use syntax like zeros(10, dtype=numpy.float64) where you have dtypes like numpy.int8 numpy.uint32 numpy.complex128 # == two numpy.float64, one for real, one for imag. etc. This is usually less confusing to my mind. One caveat: floats above 64 bits are not always supported and rarely what you think - see another discussion on this list a few weeks back. Cheers Paul On 28. nov. 2011, at 09:10, Mads Ipsen wrote: Hi, Where can I find a complete and exhaustive description of the dtype syntax and arguments. For example, something like a = arange(4, dtype='=H') seems hard to extract from the documentation that I can find on the web and the numpy docstrings. Best regards Mads -- +-+ | Mads Ipsen | +--+--+ | Gåsebæksvej 7, 4. tv | | | DK-2500 Valby| phone: +45-29716388 | | Denmark | email: mads.ip...@gmail.com | +--+--+ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.savetxt for complex arrays
Hello numpy fans, a patch and pull request for ticket 1573 has been posted, giving numpy.savetxt the possibility to save complex arrays to text files. Formatting of the output is supported roughly along the same lines as for real numbers. https://github.com/numpy/numpy/pull/172 Share, test, discuss, and enjoy! Cheers Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] build errors
You can use the BLAS and LAPACK environment variables. export BLAS=/path/to/libatlas.so export LAPACK=/path/to/libatlas.so python setup.py build I've recently had problems with ATLAS solving equation systems incorrectly for certain inputs with no adequate explanation. Re-running the same simulation but linking to Goto2 BLAS/LAPACK instead solved the problem. Strange, yet worrisome. YMMV! Cheers Paul On Thu, Oct 6, 2011 at 3:05 PM, Miah Wadud Dr (ITCS) w.m...@uea.ac.uk wrote: Hi again, I have built the ATLAS dynamic shared libraries and now need to tell numpy to build against them which are located in a different location to where it expects them. Do you know how I can do that? The command I am using to build numpy is: python setup.py build --fcompiler=gnu95 but this looks in /usr/lib64/atlas and I need it to look in another location (/gpfs/grace/atlas-3.8.4). Thanks in advance, Regards. -Original Message- From: numpy-discussion-boun...@scipy.org [mailto:numpy-discussion- boun...@scipy.org] On Behalf Of Miah Wadud Dr (ITCS) Sent: Thursday, October 06, 2011 11:12 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] build errors Hi David, Thanks for your reply. Nope, I didn't build the ATLAS libraries myself and am trying to do that now. However, whenever I try to build the shared libraries using the configure command: [root@cn130 linux]# ../configure -Fa alg -fPIC --prefix=/gpfs/grace/atlas-3.8.4 it keeps building the static version. The ATLAS documentation stated that I need to provide the above flags to build the dynamic ones but this doesn't seem to work. Any help will be greatly appreciated. Thanks in advance. Regards, Wadud. -Original Message- From: numpy-discussion-boun...@scipy.org [mailto:numpy-discussion- boun...@scipy.org] On Behalf Of David Cournapeau Sent: Tuesday, October 04, 2011 6:40 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] build errors On Tue, Oct 4, 2011 at 11:54 AM, Miah Wadud Dr (ITCS) w.m...@uea.ac.uk wrote: Hello numpy users, I am trying to build numpy 1.6.1 and am having problems. It prints the following error message: gcc -pthread -shared build/temp.linux-x86_64- 2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib64/atlas -Lbuild/temp.linux- x86_64- 2.4 -lptf77blas -lptcblas -latlas -o build/lib.linux-x86_64- 2.4/numpy/core/_dotblas.so /usr/bin/ld: /usr/lib64/atlas/libptcblas.a(cblas_dptgemm.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC /usr/lib64/atlas/libptcblas.a: could not read symbols: Bad value collect2: ld returned 1 exit status /usr/bin/ld: /usr/lib64/atlas/libptcblas.a(cblas_dptgemm.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC /usr/lib64/atlas/libptcblas.a: could not read symbols: Bad value collect2: ld returned 1 exit status error: Command gcc -pthread -shared build/temp.linux-x86_64- 2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib64/atlas -Lbuild/temp.linux- x86_64- 2.4 -lptf77blas -lptcblas -latlas -o build/lib.linux-x86_64- 2.4/numpy/core/_dotblas.so failed with exit status 1 Did you build Atlas by yourself ? If so, it is most likely not usable for shared libraries (mandatory for any python extension, including bumpy). You need to configure atlas with the option -Fa alg -fPIC. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Eigenvalues did not converge
I'm not sure if I got my point across. My point was that the ATLAS installation that numpy linked against was broken on Mac OS X but not on Linux (afaik). Hence, your code may run better on your supercomputer. So, try linking against a different BLAS/LAPACK implementation, and, with some luck, your problem could potentially disappear. ymmv, Paul. On Thu, Sep 1, 2011 at 8:20 PM, Rick Muller rpmul...@gmail.com wrote: Yes, as I pointed out, the problem does run on the Macintosh systems. But I'd like to be able to run these on our linux supercomputers. Surely this is possible, right? On Mon, Aug 29, 2011 at 9:31 AM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: I recently got into trouble with these calculations (although I used scipy). I actually got segfaults and bus errors. The solution for me was to not link against ATLAS, but rather link against Apple's blas/lapack libraries. That got everything working again. I would suggest trying to install against something other than ATLAS and see if that helps (or, more generally, determining which blas/lapack you are linking against, and try something else). Paul On 29. aug. 2011, at 16.21, Charanpal Dhanjal wrote: I posted a similar question about the non-convergence of numpy.linalg.svd a few weeks ago. I'm not sure I can help but I wonder if you compiled numpy with ATLAS/MKL support (try numpy.show_config()) and whether it made a difference? Also what is the condition number and Frobenius norm of the matrix in question? Charanpal On Mon, 29 Aug 2011 08:56:31 -0600, Rick Muller wrote: Im bumping into the old Eigenvalues did not converge error using numpy.linalg.eigh() on several different linux builds of numpy (1.4.1). The matrix is 166x166. I can compute the eigenvalues on a Macintosh build of numpy, and I can confirm that there arent degenerate eigenvalues, and that the matrix appears to be negative definite. Ive seen this before (though not for several years), and what I normally do is to build lapack with -O0. This trick did not work in the current instance. Does anyone have any tricks to getting eigh to work? Other weird things that Ive noticed about this case: I can compute the eigenvalues using eigvals and eigvalsh, and can compute the eigenvals/vecs using eig(). The matrix is real symmetric, and Ive tested that its symmetric enough by forcibly symmetrizing it. Thanks in advance for any help you can offer. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Rick Muller rpmul...@gmail.com 505-750-7557 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Eigenvalues did not converge
I recently got into trouble with these calculations (although I used scipy). I actually got segfaults and bus errors. The solution for me was to not link against ATLAS, but rather link against Apple's blas/lapack libraries. That got everything working again. I would suggest trying to install against something other than ATLAS and see if that helps (or, more generally, determining which blas/lapack you are linking against, and try something else). Paul On 29. aug. 2011, at 16.21, Charanpal Dhanjal wrote: I posted a similar question about the non-convergence of numpy.linalg.svd a few weeks ago. I'm not sure I can help but I wonder if you compiled numpy with ATLAS/MKL support (try numpy.show_config()) and whether it made a difference? Also what is the condition number and Frobenius norm of the matrix in question? Charanpal On Mon, 29 Aug 2011 08:56:31 -0600, Rick Muller wrote: Im bumping into the old Eigenvalues did not converge error using numpy.linalg.eigh() on several different linux builds of numpy (1.4.1). The matrix is 166x166. I can compute the eigenvalues on a Macintosh build of numpy, and I can confirm that there arent degenerate eigenvalues, and that the matrix appears to be negative definite. Ive seen this before (though not for several years), and what I normally do is to build lapack with -O0. This trick did not work in the current instance. Does anyone have any tricks to getting eigh to work? Other weird things that Ive noticed about this case: I can compute the eigenvalues using eigvals and eigvalsh, and can compute the eigenvals/vecs using eig(). The matrix is real symmetric, and Ive tested that its symmetric enough by forcibly symmetrizing it. Thanks in advance for any help you can offer. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] saving groups of numpy arrays to disk
On 25. aug. 2011, at 23.49, David Warde-Farley wrote: On 2011-08-25, at 2:42 PM, Chris.Barker wrote: On 8/24/11 9:22 AM, Anthony Scopatz wrote: You can use Python pickling, if you do *not* have a requirement for: I can't recall why, but it seem pickling of numpy arrays has been fragile and not very performant. I like the npy / npz format, built in to numpy, if you don't need: - access from non-Python programs While I'm not aware of reader implementations for any other language, NPY is a dirt-simple and well-documented format designed by Robert Kern, and should be readable without too much trouble from any language that supports binary I/O. The full spec is at https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt It should be especially trivial to read arrays of simple scalar numeric dtypes, but reading compound dtypes is also doable. For NPZ, use a standard zip file reading library to access individual files in the archive, which are in .npy format (or just unzip it by hand first -- it's a normal .zip file with a special extension). David Out of curiosity: is the .npy format guaranteed to be independent of architecture (endianness and similar issues)? Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] saving groups of numpy arrays to disk
Hi! On 21. aug. 2011, at 00.18, Chris Withers wrote: Hi All, I've got a tree of nested dicts that at their leaves end in numpy arrays of identical sizes. What's the easiest way to persist these to disk so that I can pick up with them where I left off? Probably giving them names like trunk_branch_leaf.txt with numpy.savetxt, if you want it quick and dirty. Or possibly, use numpy.savez directly on your dict. What's the most correct way to do so? I'm using IPython if that makes things easier... I had wondered about PyTables, but that seems a bit too heavyweight for this, unless I'm missing something? In my (perhaps limited) experience, hdf5 is great for this. I personally use h5py, I believe it is a little lighter. You get the tree structure for free in something like a directory structure: /branch/leaf /trunk/branch/leaf etc. Cheers Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reconstruct multidimensional array from buffer without shape
On 19. aug. 2011, at 19.57, Ian wrote: Right. I'm new to NumPy so I figured I'd check if there was some nifty way of preserving the shape without storing it in the database that I hadn't discovered yet. No worries, I'll store the shape alongside the array. Thanks for the reply. I love the h5py package so I keep recommending it (and pytables is supposed to be good, I think?). h5py stores files in hdf5, which is readable from C,C++,fortran,java,python... It also keeps track of shape and you can store other metadata (e.g. strings) as desired. Also I believe the numpy format (see e.g. http://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html#numpy.savez) can do the same, although I don't think performance scales as well for huge arrays, and it's not language-neutral (to my knowledge). Cheers Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] bug with latest numpy git snapshot build with Python3
On 13. aug. 2011, at 20.42, Charles R Harris wrote: 2011/8/11 Dmitrey tm...@ukr.net bug in KUBUNTU 11.04, latest numpy git snapshot build with Python3 import numpy Traceback (most recent call last): File stdin, line 1, in module File /usr/local/lib/python3.2/dist-packages/numpy/__init__.py, line 137, in module from . import add_newdocs File /usr/local/lib/python3.2/dist-packages/numpy/add_newdocs.py, line 9, in module from numpy.lib import add_newdoc File /usr/local/lib/python3.2/dist-packages/numpy/lib/__init__.py, line 4, in module from .type_check import * File /usr/local/lib/python3.2/dist-packages/numpy/lib/type_check.py, line 8, in module import numpy.core.numeric as _nx File /usr/local/lib/python3.2/dist-packages/numpy/core/__init__.py, line 10, in module from .numeric import * File /usr/local/lib/python3.2/dist-packages/numpy/core/numeric.py, line 27, in module import multiarray ImportError: No module named multiarray I don't see this. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Mac OS X 10.6.8, python3.2, I don't see this either. import multiarray does not work, but import numpy works beautifully. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Efficient way to load a 1Gb file?
On 10. aug. 2011, at 21.03, Gael Varoquaux wrote: On Wed, Aug 10, 2011 at 04:01:37PM -0400, Anne Archibald wrote: A 1 Gb text file is a miserable object anyway, so it might be desirable to convert to (say) HDF5 and then throw away the text file. +1 G +1 and a very warm recommendation of h5py. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.savetxt Ticket 1573 - suggested fix
(A pull request has been submitted on github, but I'm posting here so people can discuss the user interface issues.) As of now, the fmt= kwarg kan be (for complex dtype): a) a single specifier, fmt='%.4e', resulting in numbers formatted like ' (%s+%sj)' % (fmt, fmt) b) a full string specifying every real and imaginary part, e.g. ' %.4e %+.4j' * 3 for 3 columns c) a list of specifiers, one per column - in this case, the real and imaginary part must have separate specifiers, e.g. ['%.3e + %.3ej', '(%.15e%+.15ej)'] It would be good if people could air their opinion as to whether this is what they would expect from savetxt behavior for real (float) numbers. Ticket link: http://projects.scipy.org/numpy/ticket/1573 Cheers, Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Cython 0.15
Looks like you have done some great work! I've been using f2py in the past, but I always liked the idea of cython - gradually wrapping more and more code as the need arises. I read somewhere that fortran wrapping with cython was coming - dare I ask what the status on this is? Is it a goal for cython to support easy fortran wrapping at all? Keep up the good work! Paul On 6. aug. 2011, at 11.18, Dag Sverre Seljebotn wrote: We are excited to announce the release of Cython 0.15, which is a huge step forward in achieving full Python language coverage as well as many new features, optimizations, and bugfixes. Download: http://cython.org/ or http://pypi.python.org/pypi/Cython == Major Features == * Generators (yield) - Cython has full support for generators, generator expressions and coroutines. http://www.python.org/dev/peps/pep-0342/ * The nonlocal keyword is supported. * Re-acquiring the gil: with gil - works as expected within a nogil context. * OpenMP support: http://docs.cython.org/0.15/src/userguide/parallelism.html * Control flow analysis prunes dead code and emits warnings and errors about uninitialised variables. * Debugger command cy set to assign values of expressions to Cython variables and cy exec counterpart cy_eval(). * Exception chaining http://www.python.org/dev/peps/pep-3134/ * Relative imports http://www.python.org/dev/peps/pep-0328/ * The with statement has its own dedicated and faster C implementation. * Improved pure syntax including cython.cclass, cython.cfunc, and cython.ccall. http://docs.cython.org/0.15/src/tutorial/pure.html * Support for del. * Boundschecking directives implemented for builtin Python sequence types. * Several updates and additions to the shipped standard library pxd files https://github.com/cython/cython/tree/master/Cython/Includes * Forward declaration of types is no longer required for circular references. Note: this will be the last release to support Python 2.3; Python 2.4 will be supported for at least one more release. == General improvements and bug fixes == This release contains over a thousand commits including hundreds of bugfixes and optimizations. The bug tracker has not been as heavily used this release cycle, but is still useful http://trac.cython.org/cython_trac/query?status=closedgroup=componentorder=idcol=idcol=summarycol=milestonecol=statuscol=typecol=prioritycol=ownercol=componentmilestone=0.15desc=1 == Incompatible changes == * Uninitialized variables are no longer initialized to None and accessing them has the same semantics as standard Python. * globals() now returns a read-only dict of the Cython module's globals, rather than the globals of the first non-Cython module in the stack * Many C++ exceptions are now special cases to give closer Python counterparts. This means that except+ functions that formally raised generic RuntimeErrors may raise something else such as ArithmaticError. == Known regressions == * The inlined generator expressions (introduced in Cython 0.13) were disabled in favour of full generator expression support. This induces a performance regression for cases that were previously inlined. == Contributors == Many thanks to: Francesc Alted, Haoyu Bai, Stefan Behnel, Robert Bradshaw, Lars Buitinck, Lisandro Dalcin, John Ehresman, Mark Florisson, Christoph Gohlke, Jason Grout, Chris Lasher, Vitja Makarov, Brent Pedersen, Dag Sverre Seljebotn, Nathaniel Smith, and Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ticket 1793
Hi! In my quest for a bug-free numpy I have, I think, fixed ticket 1793. https://github.com/numpy/numpy/pull/123 Enjoy - feedback welcome, of course. Cheers, Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py and openmp on mac os x with gfortran
Hi, I had the same problem. I think this might work: FFLAGS='-fopenmp' f2py -c (etc) The thing is that f2py doesn't let you pass the -fopenmp flag at the right time to the compiler, so you have to use some sort of environment variable trick. By the way, as far as I know, this is the case also on non-mac platforms. Did that do the trick? Cheers, Paul. On 20. juli 2011, at 21.02, Brandt Belson wrote: Hello, I'm struggling to create openmp subroutines. I've simplified the problem down to the subroutine below. -- play.f90 -- subroutine step(soln,n) implicit none integer n,i real*8 soln(n) !f2py intent(in) n !f2py intent(out) soln !f2py depend(n) soln !$OMP PARALLEL DO do i=1,n soln(i) = .1 end do !$OMP END PARALLEL DO end subroutine step I compile this with the command: f2py -c -m play play.f90 --fcompiler=gfortran --f90flags=-fopenmp This completes successfully. When I import the module, I get the following error message. $ python -c 'import play' Traceback (most recent call last): File string, line 1, in module ImportError: dlopen(./play.so, 2): Symbol not found: _GOMP_parallel_end Referenced from: /home/bbelson/Desktop/SOR/play.so Expected in: flat namespace in /home/bbelson/Desktop/SOR/play.so It seems to me that the linking step is broken, however I did not see any other options in the f2py documentation to change the linking step. Did I miss something? Thanks, Brandt ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] silly user wish
Dear numpy developers, users, and fans: I wish for a numpy that will magically detect all my errors. Barring the invention of a numpy version which knows what I need, not what I ask of it, I have the following, simpler wish: if one takes myarray.real or myarray.imag of a float (or possibly integer), give a warning saying well that didn't make much sense, did it?. (Maybe there is some good reason why this is a bad idea, but the idea has been presented to you, nonetheless.) Best regards Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] bug in numpy.ndarray?
Hi, it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the problem. % python testcrash.py [14:13:27 on 11-05-08] type 'numpy.ndarray' [ 12.+0.1j] type 'numpy.ndarray' [ 1.+0.1j] complex128 Traceback (most recent call last): File testcrash.py, line 11, in module A[0] = A[0] + (eps1 - eps2) TypeError: can't convert complex to float % cat testcrash.py #!/usr/bin/env python import numpy A = numpy.zeros(10, dtype=numpy.complex128) eps1 = numpy.complex128([12.0 + 0.1j]) eps2 = numpy.complex128([1.0 + 0.1j]) print type(eps1), eps1 print type(eps2), eps2 print A.dtype A[0] = A[0] + (eps1 - eps2) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] bug in numpy.ndarray?
On 8. mai 2011, at 17.32, Warren Weckesser wrote: On Sun, May 8, 2011 at 7:23 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Sun, May 8, 2011 at 3:15 PM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: Hi, it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the problem. % python testcrash.py [14:13:27 on 11-05-08] type 'numpy.ndarray' [ 12.+0.1j] type 'numpy.ndarray' [ 1.+0.1j] complex128 Traceback (most recent call last): File testcrash.py, line 11, in module A[0] = A[0] + (eps1 - eps2) TypeError: can't convert complex to float % cat testcrash.py #!/usr/bin/env python import numpy A = numpy.zeros(10, dtype=numpy.complex128) eps1 = numpy.complex128([12.0 + 0.1j]) eps2 = numpy.complex128([1.0 + 0.1j]) It's the brackets, numpy.complex128([1.0 + 0.1j]) is a 1d array, not a scalar. The error message is less than helpful though. But the same pattern works fine with float64: In [2]: x = array([1.0, 2.0]) In [3]: y = array([10.0]) In [4]: x[0] = y # Works In [5]: a = array([1.0, 2.0], dtype=complex128) In [6]: b = array([10.0 + 1j]) In [7]: a[0] = b # Error --- TypeError Traceback (most recent call last) /Users/warren/ipython console in module() TypeError: can't convert complex to float Something is fishy about that. I agree. One thing is if arrays don't support this kind of assignment, but behavior should be the same for complex and float. IMHO. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] loadtxt ndmin option
On 5. mai 2011, at 08.49, Benjamin Root wrote: On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: On 4. mai 2011, at 20.33, Benjamin Root wrote: On Wed, May 4, 2011 at 7:54 PM, Derek Homeier de...@astro.physik.uni-goettingen.de wrote: On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it will reintroduce the 'transposed' problem? Yes, good point, one could replace the X.shape = (X.size, ) with X = np.atleast_1d(X), but for the ndmin=2 case, we'd need to replace X.shape = (X.size, 1) with X = np.atleast_2d(X).T - not sure which solution is more efficient in terms of memory access etc... Cheers, Derek I can confirm that the current behavior is not sufficient for all of the original corner cases that ndmin was supposed to address. Keep in mind that np.loadtxt takes a one-column data file and a one-row data file down to the same shape. I don't see how the current code is able to produce the correct array shape when ndmin=2. Do we have some sort of counter in loadtxt for counting the number of rows and columns read? Could we use those to help guide the ndmin=2 case? I think that using atleast_1d(X) might be a bit overkill, but it would be very clear as to the code's intent. I don't think we have to worry about memory usage if we limit its use to only situations where ndmin is greater than the number of dimensions of the array. In those cases, the array is either an empty result, a scalar value (in which memory access is trivial), or 1-d (in which a transpose is cheap). What if one does things the other way around - avoid calling squeeze until _after_ doing the atleast_Nd() magic? That way the row/column information should be conserved, right? Also, we avoid transposing, memory use, ... Oh, and someone could conceivably have a _looong_ 1D file, but would want it read as a 2D array. Paul @Derek, good catch with noticing the error in the tests. We do still need to handle the case I mentioned, however. I have attached an example script to demonstrate the issue. In this script, I would expect the second-to-last array to be a shape of (1, 5). I believe that the single-row, multi-column case would actually be the more common type of edge-case encountered by users than the others. Therefore, I believe that this ndmin fix is not adequate until this is addressed. @Paul, we can't call squeeze after doing the atleast_Nd() magic. That would just undo whatever we had just done. Also, wrt the transpose, a (1, 10) array looks the same in memory as a (10, 1) array, right? Agree. I thought more along the lines of (pseudocode-ish) if ndmin == 0: squeeze() if ndmin == 1: atleast_1D() elif ndmin == 2: atleast_2D() else: I don't rightly know what would go here, maybe raise ValueError? That would avoid the squeeze call before the atleast_Nd magic. But the code was changed, so I think my comment doesn't make sense anymore. It's probably fine the way it is! Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] loadtxt ndmin option
On 4. mai 2011, at 17.34, Derek Homeier wrote: Hi Paul, I've got back to your suggestion re. the ndmin flag for loadtxt from a few weeks ago... On 27.03.2011, at 12:09PM, Paul Anton Letnes wrote: 1562: I attach a possible patch. This could also be the default behavior to my mind, since the function caller can simply call numpy.squeeze if needed. Changing default behavior would probably break old code, however. See comments on Trac as well. Your patch is better, but there is one thing I disagree with. 808if X.ndim ndmin: 809if ndmin == 1: 810X.shape = (X.size, ) 811elif ndmin == 2: 812X.shape = (X.size, 1) The last line should be: 812X.shape = (1, X.size) If someone wants a 2D array out, they would most likely expect a one-row file to come out as a one-row array, not the other way around. IMHO. I think you are completely right for the test case with one row. More generally though, since a file of N rows and M columns is read into an array of shape (N, M), ndmin=2 should enforce X.shape = (1, X.size) for single-row input, and X.shape = (X.size, 1) for single-column input. I thought this would be handled automatically by preserving the original 2 dimensions, but apparently with single-row/multi-column input an extra dimension 1 is prepended when the array is returned from the parser. I've put up a fix for this at https://github.com/dhomeier/numpy/compare/master...ndmin-cols and also tested the patch against 1.6.0.rc2. Cheers, Derek Looks sensible to me at least! But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it will reintroduce the 'transposed' problem? Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] loadtxt ndmin option
On 4. mai 2011, at 20.33, Benjamin Root wrote: On Wed, May 4, 2011 at 7:54 PM, Derek Homeier de...@astro.physik.uni-goettingen.de wrote: On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it will reintroduce the 'transposed' problem? Yes, good point, one could replace the X.shape = (X.size, ) with X = np.atleast_1d(X), but for the ndmin=2 case, we'd need to replace X.shape = (X.size, 1) with X = np.atleast_2d(X).T - not sure which solution is more efficient in terms of memory access etc... Cheers, Derek I can confirm that the current behavior is not sufficient for all of the original corner cases that ndmin was supposed to address. Keep in mind that np.loadtxt takes a one-column data file and a one-row data file down to the same shape. I don't see how the current code is able to produce the correct array shape when ndmin=2. Do we have some sort of counter in loadtxt for counting the number of rows and columns read? Could we use those to help guide the ndmin=2 case? I think that using atleast_1d(X) might be a bit overkill, but it would be very clear as to the code's intent. I don't think we have to worry about memory usage if we limit its use to only situations where ndmin is greater than the number of dimensions of the array. In those cases, the array is either an empty result, a scalar value (in which memory access is trivial), or 1-d (in which a transpose is cheap). What if one does things the other way around - avoid calling squeeze until _after_ doing the atleast_Nd() magic? That way the row/column information should be conserved, right? Also, we avoid transposing, memory use, ... Oh, and someone could conceivably have a _looong_ 1D file, but would want it read as a 2D array. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2
On Mac OS X 10.6.7 with macports python 2.7 I get as follows: OK (KNOWNFAIL=3, SKIP=1) Python version: Python 2.7.1 (r271:86832, Jan 12 2011, 10:44:27) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin % gfortran --version GNU Fortran (GCC) 4.5.3 Well done to everyone working on numpy 1.6! Paul. On 3. mai 2011, at 11.18, Ralf Gommers wrote: Hi, I am pleased to announce the availability of the second release candidate of NumPy 1.6.0. Compared to the first release candidate, one segfault on (32-bit Windows + MSVC) and several memory leaks were fixed. If no new problems are reported, the final release will be in one week. Sources and binaries can be found at http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ For (preliminary) release notes see below. Enjoy, Ralf = NumPy 1.6.0 Release Notes = This release includes several new features as well as numerous bug fixes and improved documentation. It is backward compatible with the 1.5.0 release, and supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights == * Re-introduction of datetime dtype support to deal with dates in arrays. * A new 16-bit floating point type. * A new iterator, which improves performance of many functions. New features New 16-bit floating point type -- This release adds support for the IEEE 754-2008 binary16 format, available as the data type ``numpy.half``. Within Python, the type behaves similarly to `float` or `double`, and C extensions can add support for it with the exposed half-float API. New iterator A new iterator has been added, replacing the functionality of the existing iterator and multi-iterator with a single object and API. This iterator works well with general memory layouts different from C or Fortran contiguous, and handles both standard NumPy and customized broadcasting. The buffering, automatic data type conversion, and optional output parameters, offered by ufuncs but difficult to replicate elsewhere, are now exposed by this iterator. Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` - Extend the number of polynomials available in the polynomial package. In addition, a new ``window`` attribute has been added to the classes in order to specify the range the ``domain`` maps to. This is mostly useful for the Laguerre, Hermite, and HermiteE polynomials whose natural domains are infinite and provides a more intuitive way to get the correct mapping of values without playing unnatural tricks with the domain. Fortran assumed shape array and size function support in ``numpy.f2py`` --- F2py now supports wrapping Fortran 90 routines that use assumed shape arrays. Before such routines could be called from Python but the corresponding Fortran routines received assumed shape arrays as zero length arrays which caused unpredicted results. Thanks to Lorenz Hüdepohl for pointing out the correct way to interface routines with assumed shape arrays. In addition, f2py interprets Fortran expression ``size(array, dim)`` as ``shape(array, dim-1)`` which makes it possible to automatically wrap Fortran routines that use two argument ``size`` function in dimension specifications. Before users were forced to apply this mapping manually. Other new functions --- ``numpy.ravel_multi_index`` : Converts a multi-index tuple into an array of flat indices, applying boundary modes to the indices. ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose the underlying type promotion used by the ufuncs and other operations to determine the types of outputs. These improve upon the ``numpy.common_type`` and ``numpy.mintypecode`` which provide similar functionality but do not match the ufunc implementation. Changes === Changes and improvements in the numpy core -- ``default error handling`` -- The default error handling has been change from ``print`` to ``warn`` for all except for ``underflow``, which remains as ``ignore``. ``numpy.distutils`` --- Several new compilers are supported for building Numpy: the Portland Group Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C compiler on Linux. ``numpy.testing`` - The testing
Re: [Numpy-discussion] Zero row in SVD's unitary matrix on some Mac's
On 25. apr. 2011, at 19.57, Pauli Virtanen wrote: On Mon, 25 Apr 2011 10:16:13 -0700, Rob Beezer wrote: [clip] Many more details and complete transcripts are at: http://trac.sagemath.org/sage_trac/ticket/11248 Any thoughts or advice to help us understand this would be greatly appreciated. The Numpy routine is a very thin wrapper of LAPACK's ZGESDD, and probably cannot have any bugs of this kind, so the problem is most likely with the LAPACK and BLAS libraries you use. You will probably be able to reproduce the problem also with an equivalent Fortran/C snippet calling LAPACK directly. Problems like this in BLAS/LAPACK are somewhat difficult to track. You could try switching to another BLAS library (or, if you use ATLAS, compile it differently) and checking if the problem disappears. -- Pauli Virtanen I cannot claim anything concrete, but I did notice something seemingly really odd about the blas+lapack that ships with macosx. In our light scattering simulation code we use energy conservation as a test of consistency. When running small test runs on my macbook pro, the energy would be significantly less well conserved compared to when the exact same simulation was run on a linux box. No harm done, we don't care about the exact results on the laptop, but it did seem odd. We do not rely on SVD, however, we only use LU+backsubstitution (cgesv). I have a vague feeling that there is something odd about lapack+blas on macosx, but I do not have any hard evidence. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Strange behavior of operator *=
On 5. apr. 2011, at 15.39, Alan G Isaac wrote:
On 4/5/2011 9:26 AM, François Steinmetz wrote:
It does not change the dtype, 'int' is just interpreted as 'int64' :
So the meaning of 'int' is system specific?
import numpy as np; a=np.eye(2,dtype='int'); a.dtype
dtype('int32')
Alan Isaac
'int' is system specific, 'int32' and 'int64' is not. At least, as far as I can
tell.
Paul.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Import patch
Hi. When looking at the loadtxt/savetxt tickets, I noticed that the 're' module is imported in an odd place. I therefore suggest that this import is moved to the top of the file, in order to gather these as much as possible. I find the code easier to read then. After all, there is no 'try / catch' or similar to check if the module exists. See patch below. I do not believe any tests or tickets are needed - correct me if I am wrong. Cheers, Paul. --- a/numpy/lib/npyio.pySat Apr 02 20:19:55 2011 -0600 +++ b/numpy/lib/npyio.pySun Apr 03 12:30:02 2011 +0200 @@ -6,6 +6,7 @@ import format import sys import os +import re import sys import itertools import warnings @@ -956,7 +957,6 @@ if own_fh: fh.close() -import re def fromregex(file, regexp, dtype): Construct an array from a text file, using regular expression parsing. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Import patch
On 4. apr. 2011, at 15.34, Charles R Harris wrote: On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: Hi. When looking at the loadtxt/savetxt tickets, I noticed that the 're' module is imported in an odd place. I therefore suggest that this import is moved to the top of the file, in order to gather these as much as possible. I find the code easier to read then. After all, there is no 'try / catch' or similar to check if the module exists. See patch below. I do not believe any tests or tickets are needed - correct me if I am wrong. Cheers, Paul. --- a/numpy/lib/npyio.pySat Apr 02 20:19:55 2011 -0600 +++ b/numpy/lib/npyio.pySun Apr 03 12:30:02 2011 +0200 @@ -6,6 +6,7 @@ import format import sys import os +import re import sys import itertools import warnings @@ -956,7 +957,6 @@ if own_fh: fh.close() -import re def fromregex(file, regexp, dtype): Construct an array from a text file, using regular expression parsing. If you want to see a lot of other small things for cleanups, run pyflakes or pylint on the files in numpy/lib Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Are you suggesting that I should do this and submit one or more patches? By the way - what is a suggested form of submitting patches? Using e-mail seems a bit clumsy to me. Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Import patch
On 4. apr. 2011, at 16.42, Charles R Harris wrote: On Mon, Apr 4, 2011 at 8:29 AM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: On 4. apr. 2011, at 15.34, Charles R Harris wrote: On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: Hi. When looking at the loadtxt/savetxt tickets, I noticed that the 're' module is imported in an odd place. I therefore suggest that this import is moved to the top of the file, in order to gather these as much as possible. I find the code easier to read then. After all, there is no 'try / catch' or similar to check if the module exists. See patch below. I do not believe any tests or tickets are needed - correct me if I am wrong. Cheers, Paul. --- a/numpy/lib/npyio.pySat Apr 02 20:19:55 2011 -0600 +++ b/numpy/lib/npyio.pySun Apr 03 12:30:02 2011 +0200 @@ -6,6 +6,7 @@ import format import sys import os +import re import sys import itertools import warnings @@ -956,7 +957,6 @@ if own_fh: fh.close() -import re def fromregex(file, regexp, dtype): Construct an array from a text file, using regular expression parsing. If you want to see a lot of other small things for cleanups, run pyflakes or pylint on the files in numpy/lib Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Are you suggesting that I should do this and submit one or more patches? That would be great, and a good way to get into numpy development. By the way - what is a suggested form of submitting patches? Using e-mail seems a bit clumsy to me. The best thing would be to setup to do your work on github and then issue pull requests, see the directions here: http://docs.scipy.org/doc/numpy/dev/gitwash/development_setup.html. If you are on linux or mac getting set up should be pretty easy. Windows may be trickier, I don't have experience doing development on that platform. I'm on a mac, and I've got git. The skills to use it will have to grow with time. I submitted a pull request on this particular patch - let me know if I got it right! Cheers, Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] loadtxt/savetxt tickets
On 26. mars 2011, at 21.44, Derek Homeier wrote:
Hi Paul,
having had a look at the other tickets you dug up,
My opinions are my own, and in detail, they are:
1752:
I attach a possible patch. FWIW, I agree with the request. The
patch is written to be compatible with the fix in ticket #1562, but
I did not test that yet.
Tested, see also my comments on Trac.
Great!
1731:
This seems like a rather trivial feature enhancement. I attach a
possible patch.
Agreed. Haven't tested it though.
Great!
1616:
The suggested patch seems reasonable to me, but I do not have a
full list of what objects loadtxt supports today as opposed to what
this patch will support.
Looks like you got this one. Just remember to make it compatible with #1752.
Should be easy.
1562:
I attach a possible patch. This could also be the default
behavior to my mind, since the function caller can simply call
numpy.squeeze if needed. Changing default behavior would probably
break old code, however.
See comments on Trac as well.
Your patch is better, but there is one thing I disagree with.
808if X.ndim ndmin:
809if ndmin == 1:
810X.shape = (X.size, )
811elif ndmin == 2:
812X.shape = (X.size, 1)
The last line should be:
812X.shape = (1, X.size)
If someone wants a 2D array out, they would most likely expect a one-row file
to come out as a one-row array, not the other way around. IMHO.
1458:
The fix suggested in the ticket seems reasonable, but I have
never used record arrays, so I am not sure of this.
There were some issues with Python3, and I also had some general
reservations
as noted on Trac - basically, it makes 'unpack' equivalent to
transposing for 2D-arrays,
but to splitting into fields for 1D-recarrays. My question was, what's
going to happen
when you get to 2D-recarrays? Currently this is not an issue since
loadtxt can only
read 2D regular or 1D structured arrays. But this might change if the
data block
functionality (see below) were to be implemented - data could then be
returned as
3D arrays or 2D structured arrays... Still, it would probably make
most sense (or at
least give the widest functionality) to have 'unpack=True' always
return a list or iterator
over columns.
OK, I don't know recarrays, as I said.
1445:
Adding this functionality could break old code, as some old
datafiles may have empty lines which are now simply ignored. I do
not think the feature is a good idea. It could rather be implemented
as a separate function.
1107:
I do not see the need for this enhancement. In my eyes, the
usecols kwarg does this and more. Perhaps I am misunderstanding
something here.
Agree about #1445, and the bit about 'usecols' - 'numcols' would just
provide a
shorter call to e.g. read the first 20 columns of a file (well, not
even that much
over 'usecols=range(20)'...), don't think that justifies an extra
argument.
But the 'datablocks' provides something new, that a number of people
seem
to miss from e.g. gnuplot (including me, actually ;-). And it would
also satisfy the
request from #1445 without breaking backwards compatibility.
I've been wondering if could instead specify the separator lines
through the
parameter, e.g. blocksep=['None', 'blank','invalid'], not sure if
that would make
it more useful...
What about writing a separate function, e.g. loadblocktxt, and have it separate
the chunks and call loadtxt for each chunk? Just a thought. Another possibility
would be to write a function that would let you load a set of text files in a
directory, and return a dict of datasets, one per file. One could write a
similar save-function, too. They would just need to call loadtxt/savetxt on a
per-file basis.
1071:
It is not clear to me whether loadtxt is supposed to support
missing values in the fashion indicated in the ticket.
In principle it should at least allow you to, by the use of converters
as described there.
The problem is, the default delimiter is described as 'any
whitespace', which in the
present implementation obviously includes any number of blanks or
tabs. These
are therefore treated differently from delimiters like ',' or ''. I'd
reckon there are
too many people actually relying on this behaviour to silently change it
(e.g. I know plenty of tables with columns separated by either one or
several
tabs depending on the length of the previous entry). But the tab is
apparently also
treated differently if explicitly specified with delimiter='\t' -
and in that case using
a converter à la {2: lambda s: float(s or 'Nan')} is working for
fields in the middle of
the line, but not at the end - clearly warrants improvement. I've
prepared a patch
working for Python3 as well.
Great!
1163:
1565:
These tickets seem to have the same origin of the problem. I
attach one
Re: [Numpy-discussion] loadtxt/savetxt tickets
Hi! I have had a look at the list of numpy.loadtxt tickets. I have never contributed to numpy before, so I may be doing stupid things - don't be afraid to let me know! My opinions are my own, and in detail, they are: 1752: I attach a possible patch. FWIW, I agree with the request. The patch is written to be compatible with the fix in ticket #1562, but I did not test that yet. 1731: This seems like a rather trivial feature enhancement. I attach a possible patch. 1616: The suggested patch seems reasonable to me, but I do not have a full list of what objects loadtxt supports today as opposed to what this patch will support. 1562: I attach a possible patch. This could also be the default behavior to my mind, since the function caller can simply call numpy.squeeze if needed. Changing default behavior would probably break old code, however. 1458: The fix suggested in the ticket seems reasonable, but I have never used record arrays, so I am not sure of this. 1445: Adding this functionality could break old code, as some old datafiles may have empty lines which are now simply ignored. I do not think the feature is a good idea. It could rather be implemented as a separate function. 1107: I do not see the need for this enhancement. In my eyes, the usecols kwarg does this and more. Perhaps I am misunderstanding something here. 1071: It is not clear to me whether loadtxt is supposed to support missing values in the fashion indicated in the ticket. 1163: 1565: These tickets seem to have the same origin of the problem. I attach one possible patch. The previously suggested patches that I've seen will not correctly convert floats to ints, which I believe my patch will. I hope you find this useful! Is there some way of submitting the patches for review in a more convenient fashion than e-mail? Cheers, Paul. 1562.patch Description: Binary data 1163.patch Description: Binary data 1731.patch Description: Binary data 1752.patch Description: Binary data On 25. mars 2011, at 16.06, Charles R Harris wrote: Hi All, Could someone with an interest in loadtxt/savetxt look through the associated tickets? A search on the tickets using either of those keys will return fairly lengthy lists. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] loadtxt/savetxt tickets
Hi Derek! On 26. mars 2011, at 15.48, Derek Homeier wrote: Hi again, On 26 Mar 2011, at 15:20, Derek Homeier wrote: 1562: I attach a possible patch. This could also be the default behavior to my mind, since the function caller can simply call numpy.squeeze if needed. Changing default behavior would probably break old code, Seems the fastest solution unless someone wants to change numpy.squeeze as well. But the present patch does not call np.squeeze any more at all, so I propose to restore that behaviour for X.ndim ndmin to remain really backwards compatible. It also seems easier to code when making the default ndmin=0. I've got another somewhat general question: since it would probably be nice to have a test for this, I found one could simply add something along the lines of assert_equal(a.shape, x.shape) to test_io.py - test_shaped_dtype(self) or should one generally create a new test for such things (might still be better in this case, since test_shaped_dtype does not really test different ndim)? Cheers, Derek It would be nice to see your patch. I uploaded all of mine as mentioned. I'm no testing expert, but I am sure someone else will comment on it. Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to loop read input over various file with same extension in a folder (newbie question)
One way is to use python dicts. Pseudocode:
arrays = {}
integer = 0
for file in allfiles:
integer +=1
data = file.read()
arrays['a' + '%d' % integer] = data
I would consider using the filename as the key instead of a1, a2, etc. That way
you have that information readily available as well.
Good luck,
Paul.
On 24. mars 2011, at 08.27, Sachin Kumar Sharma wrote:
BB,
Currently I am reading a text file and storing the output as an array a1,
some arithmetic operation on it and plotting it. I want to do the same over
1000 files.
How to loop it and how to store the input in different arrays like a2, a3 and
so on.
Also advise what is the best way to TAG or assign a comment to an array to
store which file it has read the data.
Cheers
Sachin
Sachin Kumar Sharma
Senior Geomodeler
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problems building NumPy with GotoBLAS
I'm no expert, but I just pulled off the scipy+numpy+GotoBLAS2 installation. From what I gather, the Makefile for libgoto2 downloads and compiles the generic lapack from netlib. It also wraps lapack into libgoto2.so/.a. I believe the idea is as long as the BLAS implementation is fast(TM), the lapack performance will be good. To wit*, what I did was to tell numpy where libgoto2 was: env BLAS=/path/to/libgoto2.so python setup.py install Scipy also wants the path to lapack, which is wrapped inside libgoto2: env BLAS=/path/to/libgoto2.so LAPACK=/path/to/libgoto2.so python setup.py install Afterwards, I added the path to LD_LIBRARY_PATH. This was on a linux cluster, if that matters. At any rate, I can testify that it was not a big job to get numpy and scipy working with goto blas. Good luck, Paul. *) I have notes on this on a different computer, but not available right now. On Tue, Mar 22, 2011 at 10:13 AM, Giuseppe Aprea giuseppe.ap...@gmail.comwrote: Hi all, I wonder if Peter finally got Gotoblas working with numpy. I am trying with gotoblas 1.13 installed in the same way: $ ls -R .: include lib ./include: goto ./include/goto: blaswrap.h cblas.h clapack.h f2c.h ./lib: libgoto2.a libgoto2_nehalemp-r1.13.a libgoto2_nehalemp-r1.13.so libgoto2.so and numpy 1.5.1 with this site.cfg [DEFAULT] library_dirs = /usr/local/gcc/4.5.2/gcc/lib64:/usr/local/gcc/4.5.2/gcc/lib:/usr/local/gcc/4.5.2/gcc/lib32:/usr/local/gcc/4.5.2/gotoblas2/1.13/lib:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/lib:/usr/local/gcc/4.5.2/fftw/3.2.2/lib include_dirs = /usr/local/gcc/4.5.2/gcc/include:/usr/local/gcc/4.5.2/gotoblas2/1.13/include/goto:/usr/local/gcc/4.5.2/suiteSparse/3.6.0/include:/usr/local/gcc/4.5.2/fftw/3.2.2/include search_static_first = 1 [blas_opt] libraries = goto2 language = fortran [lapack_opt] libraries = goto2 language = fortran [amd] amd_libs = amd [umfpack] umfpack_libs = umfpack [fftw] libraries = fftw3 (I also tried without _opt and language = fortran); I used goto2 for lapack because I read lapack should be included in libgoto (anyway things do not change using lapack). I am quite sure the system is not using my goto-lapack stuff since every time I buid i get: building extension numpy.numarray._capi sources building extension numpy.fft.fftpack_lite sources building extension numpy.linalg.lapack_lite sources creating build/src.linux-x86_64-2.7/numpy/linalg *### Warning: Using unoptimized lapack ###* * adding 'numpy/linalg/lapack_litemodule.c' to sources.* adding 'numpy/linalg/python_xerbla.c' to sources. adding 'numpy/linalg/zlapack_lite.c' to sources. adding 'numpy/linalg/dlapack_lite.c' to sources. adding 'numpy/linalg/blas_lite.c' to sources. adding 'numpy/linalg/dlamch.c' to sources. adding 'numpy/linalg/f2c_lite.c' to sources. building extension numpy.random.mtrand sources creating build/src.linux-x86_64-2.7/numpy/random C compiler: /usr/local/gcc/4.5.2//gcc/bin/gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O1 -pthread -fPIC -march=native -mtune=native -I/usr/local/gcc/4.5.2//gcc/include -I/usr/local/gcc/4.5.2//suiteSparse/3.6.0/include -I/usr/local/gcc/4.5.2//fftw/3.2.2/include -fPIC during numpy installation (which ends successfully). Moreover I cannot see any -lgoto2 as I would have expected. Incidentally, I cannot see -lamd, -lumfpack, -lfftw3 (or any reference to amd, umfpack, fftw3) neither, although there seems to be something to handle them in system_info.py. The failure is so complete that I must have done some big mistake but I can't correct my site.cfg even after searching the internet. This seems to be one of the major discussion about this topic so I am asking here for some help, please. Is the problem related with site.cfg or with gotoblas2 installation? Is it true that gotoblas2 hosts a full lapack inside? thank you very much! giuseppe ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What Requires C and what is just python
On 20. mars 2011, at 16.08, Ben Smith wrote: So, in addition to my computer science work, I'm a PhD student in econ. Right now, the class is using GAUSS for almost everything. This sort of pisses me off because it means people are building libraries of code that become valueless when they graduate (because right now we get GAUSS licenses for free, but it is absurdly expensive later) -- particularly when this is the only language they know. So, I had this idea of building some command line tools to do the same things using the most basic pieces of NumPy (arrays, dot products, transpose and inverse -- that's it). And it is going great. My problem however is that I'd like to be able to share these tools but I know I'm opening up a big can of worms where I have to go around building numpy on 75 peoples computers. What I'd like to do is limit myself to just the functions that are implemented in python, package it with py2exe and hand that to anyone that needs it. So, my question, if anyone knows, what's implemented in python and what depends on the c libraries? Is this even possible? I can testify that on most windows computers python(x,y) will give you everything you need - numpy, scipy, matplotlib, pyqt for GUI design, and much more. The only problem I ever saw was that some people had problems with $PATH not being set properly on windows. But this was on machines that seemed to be full of other problems. Oh, and in my experience, it is easier to run python scripts from the generic windows command line than in the ipython shell. Good luck, Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] *= operator not intuitive
Hi! This little snippet of code tricked me (in a more convoluted form). The *= operator does not change the datatype of the left hand side array. Is this intentional? It did fool me and throw my results quite a bit off. I always assumed that 'a *= b' means exactly the same as 'a = a * b' but this is clearly not the case! Paul. ++ from numpy import * a = arange(10) b = linspace(0,1,10) a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) b array([ 0., 0., 0., 0., 0., 0.5556, 0.6667, 0.7778, 0.8889, 1.]) a * b array([ 0., 0., 0., 1., 1.7778, 2.7778, 4., 5., 7., 9.]) a *= b a array([0, 0, 0, 1, 1, 2, 4, 5, 7, 9]) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] *= operator not intuitive
Heisann!
On 16. mars 2011, at 14.30, Dag Sverre Seljebotn wrote:
On 03/16/2011 02:24 PM, Paul Anton Letnes wrote:
Hi!
This little snippet of code tricked me (in a more convoluted form). The *=
operator does not change the datatype of the left hand side array. Is this
intentional? It did fool me and throw my results quite a bit off. I always
assumed that 'a *= b' means exactly the same as 'a = a * b' but this is
clearly not the case!
In [1]: a = np.ones(5)
Here, a is numpy.float64:
numpy.ones(5).dtype
dtype('float64')
In [2]: b = a
In [3]: c = a * 2
In [4]: b
Out[4]: array([ 1., 1., 1., 1., 1.])
In [5]: a *= 2
So since a is already float, and b is the same object as a, the resulting a and
b are of course floats.
In [6]: b
Out[6]: array([ 2., 2., 2., 2., 2.])
This is not the case I am describing, as in my case, a was of dtype integer.
Or did I miss something?
Paul.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] *= operator not intuitive
On 16. mars 2011, at 15.49, Chris Barker wrote: On 3/16/11 6:34 AM, Charles R Harris wrote: On Wed, Mar 16, 2011 at 7:24 AM, Paul Anton Letnes Yes, it is intentional. Numpy is more C than Python in this case, I don't know that C has anything to do with it -- the *= operators were added specifically to be in-place operators -- otherwise they would be nothing but syntactic sugar. And IIRC, numpy was one of the motivators. IMHO, the mistake was even allowing += and friends for immutables, as that inherently means something different. Of course, using += with integers is probably the most common case. -Chris I see. In that case, I have the following either/or christmas wish: Either: implement a warning along the following lines: from numpy import * a = zeros(10, dtype=complex) a.astype(float) /Users/paulanto/Library/Python/2.7/bin/bpython:2: ComplexWarning: Casting complex values to real discards the imaginary part # EASY-INSTALL-ENTRY-SCRIPT: 'bpython==0.9.7.1','console_scripts','bpython' array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) Or: give me a hint how and where to change the numpy code, and I could try to write a patch. Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] *= operator not intuitive
On 16. mars 2011, at 15.57, Dag Sverre Seljebotn wrote:
On 03/16/2011 02:35 PM, Paul Anton Letnes wrote:
Heisann!
Hei der,
On 16. mars 2011, at 14.30, Dag Sverre Seljebotn wrote:
On 03/16/2011 02:24 PM, Paul Anton Letnes wrote:
Hi!
This little snippet of code tricked me (in a more convoluted form). The *=
operator does not change the datatype of the left hand side array. Is this
intentional? It did fool me and throw my results quite a bit off. I always
assumed that 'a *= b' means exactly the same as 'a = a * b' but this is
clearly not the case!
In [1]: a = np.ones(5)
Here, a is numpy.float64:
numpy.ones(5).dtype
dtype('float64')
In [2]: b = a
In [3]: c = a * 2
In [4]: b
Out[4]: array([ 1., 1., 1., 1., 1.])
In [5]: a *= 2
So since a is already float, and b is the same object as a, the resulting a
and b are of course floats.
In [6]: b
Out[6]: array([ 2., 2., 2., 2., 2.])
This is not the case I am describing, as in my case, a was of dtype integer.
Or did I miss something?
I was just trying to demonstrate that it is NOT the case that a = a * 2
is exactly the same as a *= 2. If you assume that the two statements
are the same, then it does not make sense that b = [1, 1, ...] the first
time around, but b = [2, 2, 2...] the second time around. And in trying
to figure out why that happened, perhaps you'd see how it all fits
together...
OK, it perhaps wasn't a very good explanation...
Dag Sverre
I see, I misunderstood your point. That's another interesting aspect of this,
though.
Paul.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] *= operator not intuitive
This comes up for discussion on a fairly regular basis. I tend towards the more warnings side myself, but you aren't going to get the current behavior changed unless you can convince a large bunch of people that it is the right thing to do, which won't be easy. For one thing, a lot of current code in the wild would begin to raise warnings that weren't there before. That could also be a good thing for locating bugs, right? Or: give me a hint how and where to change the numpy code, and I could try to write a patch. You have to go down to the C level to deal with this. I guess code containing such warnings must exist in other parts of the numpy library? Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Variable in an array name?
On 12. jan. 2011, at 16.40, dstaley wrote:
Zachary Pincus-2 wrote:
Is it possible to use a variable in an array name? I am looping
through a
bunch of calculations, and need to have each array as a separate
entity.
I'm pretty new to python and numpy, so forgive my ignorance. I'm
sure there
is a simple answer, but I can't seem to find it.
let's say i have a variable 'i':
i = 5
I would like my array to have the name array5
I know how I could do this manually, but not in a loop where i is
redefined
several times.
There are ways to do this, but what you likely actually want is just
to put several arrays in a python list and then index into the list,
instead of constructing numbered names.
e.g.:
array_list = []
for whatever:
array_list.append(numpy.array(whatever))
for array in array_list:
do_something(array)
given_array = array_list[i]
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Thank you very much for the prompt response. I have already done what you
have suggested, but there are a few cases where I do need to have an array
named with a variable (looping through large numbers of unrelated files and
calculations that need to be dumped into different analyses). It would be
extraordinarily helpful if someone could post a solution to this problem,
regardless of inefficiency of the method. Thanks a ton for any additional
help.
--
This may be obvious, but I sometimes forget myself: have you tried python dicts?
from numpy import *
a = linspace(0,10)
b = a.copy()
d = {'array1':a, 'array2':b}
for key in d:
... dosomething(d[key])
That way, you can assign a name / key for each array variable, and use this
name for file names, or whatever you need names for.
Cheers
Paul.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [PATCH] gfortran under macports
On 3. des. 2010, at 16.24, Fabian Pedregosa wrote: Hi all. Macports installs gfortran as part of the gcc package, but names it gfortran-mp-$version, without providing a symbolic link to a default gcfortran executable, and thus numpy.distutils is unable to find the right executable. The attached patch very simple, it just extends possible_executables with those names, but makes the build of scipy work without having to restore to obscure fc_config flags. Fabian. 0001-FIX-recognize-macports-gfortran-compiler.patch___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Correct me if I am wrong here: If you run (sudo) gcc_select gfortran-mp-XY, where XY are the version numbers (e.g. 45 for gfortran 4.5), you should get symbolic links for the selected gcc/gfortran version. I believe that macports should probably make this clearer, and perhaps automatically when you do a port install gccXY, but I am not sure if this needs any patching? Again, I might be wrong on this. Cheers Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [PATCH] gfortran under macports
Mabe I am wrong somehow, but in my experience the easiest install of scipy is 'port install py26-scipy'. For new users, I do not see why one would recommend to build manually from source? Macports can do it for you, automagically... Paul 4. des.. 2010 15.04 Ralf Gommers ralf.gomm...@googlemail.com: On Sat, Dec 4, 2010 at 9:47 PM, Fabian Pedregosa fabian.pedreg...@inria.fr wrote: On Sat, Dec ... I would prefer to just document the gcc_select solution, since it solves the problem at hand. On the other hand, as installing on macports is not that trivial, I strongly feel that a sub... The most up-to-date instructions are at http://www.scipy.org/Installing_SciPy/Mac_OS_X, so those should be updated as well. That said, if a Macports section is added there should be a strong disclaimer that it is *not* the recommended way to install numpy/scipy. A good portion of the build problems reported on these lists are related to Fortran on OS X, and a specific gfortran build at http://r.research.att.com/tools/ is recommended for a reason. If a user really wants to use Macports some notes in the docs may help, but let's not give the impression that it's a good/default option for a new user. Cheers, Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] N dimensional dichotomy optimization
On Tue, Nov 23, 2010 at 3:37 AM, Sebastian Walter sebastian.wal...@gmail.com wrote: On Tue, Nov 23, 2010 at 11:17 AM, Gael Varoquaux gael.varoqu...@normalesup.org wrote: On Tue, Nov 23, 2010 at 11:13:23AM +0100, Sebastian Walter wrote: I'm not familiar with dichotomy optimization. Several techniques have been proposed to solve the problem: genetic algorithms, simulated annealing, Nelder-Mead and Powell. To be honest, I find it quite confusing that these algorithms are named in the same breath. I am confused too. But that stems from my lack of knowledge in optimization. Do you have a continuous or a discrete problem? Both. I would like to advertise a bit for genetic algorithms. In my experience, they seem to be the most powerful of the optimization techniques mentioned here. In particular, they are good at getting out of local minima, and don't really care if you are talking about integer or continuous problems. As long as you can think of a good representation and good genetic operators, you should be good! I have just a little experience with pyevolve myself, but it is really easy to test GAs with pyevolve, as you just have to make a few settings to get going. One word of warning: GA performance is very sensitive to the actual parameters you choose! Especially, you should think about mutation rates, crossover rates, selection protocols, and number of crossover points. (This list came off the top of my head...) If you have any GA questions, ask, and perhaps I can come up with an answer. Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
