### [Numpy-discussion] Debian/Ubuntu patch help (was: ANN: NumPy 1.6.2 release candidate 1)

Hi, if there's anyone wants to have a look at the above issue this week, that would be great. If there's a patch by this weekend I can create a second RC, so we can still have the final release before the end of this month (needed for Debian freeze). Otherwise a second RC won't be needed.

### Re: [Numpy-discussion] Debian/Ubuntu patch help

On 05/16/2012 09:01 PM, Ralf Gommers wrote: On Tue, May 15, 2012 at 10:35 PM, Julian Taylor jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com wrote: Hi, if there's anyone wants to have a look at the above issue this week, that would be great

### [Numpy-discussion] faster (selection based) median, 2013 edition

could still add median of medians for better worst case performance. If no blockers appear I want to fix this up and file a pull request to have this in numpy 1.8. Guidance on details of implementation in numpys C api is highly appreciated, its the first time I'm dealing with it. Cheers, Julian

### Re: [Numpy-discussion] quickselect

On 05/29/2013 06:12 AM, josef.p...@gmail.com wrote: On Tue, May 28, 2013 at 6:31 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi All, There is a PR adding quickselect to numpy as a function `partition`. Comments on name and exposure in the numpy API are welcome. I think the name

### Re: [Numpy-discussion] suggested change of behavior for interp

On 05.06.2013 16:33, Nathaniel Smith wrote: On Wed, Jun 5, 2013 at 3:16 PM, Slavin, Jonathan jsla...@cfa.harvard.edu wrote: The simplest monotonicity test that I've seen is: dx = np.diff(x) monotonic = np.all(dx 0.) or np.all(dx 0.) I expect that this is pretty fast, though I haven't

### Re: [Numpy-discussion] quickselect

On 09.06.2013 12:10, josef.p...@gmail.com wrote: On Wed, May 29, 2013 at 3:19 PM, josef.p...@gmail.com wrote: On Wed, May 29, 2013 at 12:25 PM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 05/29/2013 06:12 AM, josef.p...@gmail.com wrote: On Tue, May 28, 2013 at 6:31 PM, Charles R

### Re: [Numpy-discussion] quickselect

On 11.06.2013 14:37, Jonathan J. Helmus wrote: Julian, Since I am the author of the current percentile PR (https://github.com/numpy/numpy/pull/2970), I'm willing to try reimplementing percentile with the new partition functionality. I don't expect to have time to do this until the

### Re: [Numpy-discussion] svd + multiprocessing hangs

On 12.06.2013 18:07, Uwe Schmitt wrote: Dear all, the following code hangs on my Ubuntu machine. I use self compiled numpy 1.7.1 and Python 2.7.3 - import numpy import numpy.linalg import multiprocessing def classify(): X = numpy.random.random((10,3)) print before

### [Numpy-discussion] strided copy unroll, benchmarks needed

hi, I posted a pull with a minor change instructing the GCC compiler to unroll the strided copy loops (gcc will almost never do that on its own, not even on O3). https://github.com/numpy/numpy/pull/3429 It improves performance of these copies by 20%-50% depending on the size of the data copied

### Re: [Numpy-discussion] Seg. fault when running tests

On 15.06.2013 21:12, Charles R Harris wrote: On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser warren.weckes...@gmail.com mailto:warren.weckes...@gmail.com wrote: On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser warren.weckes...@gmail.com mailto:warren.weckes...@gmail.com

### Re: [Numpy-discussion] Seg. fault when running tests

On 15.06.2013 21:57, Warren Weckesser wrote: On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor @warren, can you please bisect the commit causing this? Here's the culprit: aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit commit

### Re: [Numpy-discussion] Seg. fault when running tests

wrote: On Sat, Jun 15, 2013 at 4:03 PM, Julian Taylor jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com wrote: On 15.06.2013 21:57, Warren Weckesser wrote: On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor

### Re: [Numpy-discussion] Seg. fault when running tests

On 15.06.2013 22:26, Charles R Harris wrote: On Sat, Jun 15, 2013 at 2:23 PM, Warren Weckesser warren.weckes...@gmail.com mailto:warren.weckes...@gmail.com wrote: Also on 2.7. The -O3 flag seems to cause the problem. Chuck I was wrong when I changed this flag. The compiler is

### Re: [Numpy-discussion] low level optimization in NumPy and minivect

On 17.06.2013 17:11, Frédéric Bastien wrote: Hi, I saw that recently Julian Taylor is doing many low level optimization like using SSE instruction. I think it is great. Last year, Mark Florisson released the minivect[1] project that he worked on during is master thesis. minivect

### [Numpy-discussion] quickselect via np.partition available in 1.8.dev

Hi, a selection algorithm [0] has now landed in the numpy development branch [1]. The function exposing it is: numpy.partition(data, kth=int/array, axis=-1, kind=introselect, order=None) Please see the docstrings on what it actually does (and report if they are confusing). Thanks to the numpy

### Re: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy

On 04.09.2013 12:05, Graeme B. Bell wrote: In my current GIS raster work I often have a situation where I generate code something like this: np.any([A4, A==2, B==5, ...]) However, np.any() is quite slow. It's possible to use np.logical_or to solve the problem, but then you

### Re: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy

-any-all.py Graeme On Sep 4, 2013, at 7:38 PM, Julian Taylor jtaylor.deb...@googlemail.com wrote: The result is 14 to 17x faster than np.any() for this use case.* any/all and boolean operations have been significantly speed up by vectorization in numpy 1.8 [0]. They are now around

### Re: [Numpy-discussion] ANN: 1.8.0b2 release.

try rebuilding everything from scratch. setup.py dependency handling is a bit dodgy with the generated files. On 09.09.2013 19:09, Frédéric Bastien wrote: I don't have CFLAGS defined. But I have iothers env variable that point to other python stuff like CPATH. But even in that case, I don't

### Re: [Numpy-discussion] strange behaviour with numpy arrays

On 11.09.2013 12:33, antlarac wrote: Hi, I have a numpy array, and I want to create another variable equal to it, to back it up for later calculations, because the first array will change. But after the first array changes, the second automatically changes to the same value. An example of what

### Re: [Numpy-discussion] 1.8.0rc1

On 30.09.2013 17:17, Charles R Harris wrote: Hi All, NumPy 1.8.0rc1 is up now on sourceforge http://sourceforge.net/projects/numpy/files/NumPy/1.8.0rc1/ .The binary builds are included except for Python 3.3 on windows, which will arrive later. Many thanks to Ralf for the binaries, and to

### Re: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1

On 01.10.2013 01:30, Charles R Harris wrote: On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke cgoh...@uci.edu mailto:cgoh...@uci.edu wrote: On 9/30/2013 3:45 PM, Charles R Harris wrote: On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke cgoh...@uci.edu

### Re: [Numpy-discussion] Efficient square distance computation

your computation is symmetric so you only need to compute the upper or lower triangle which will save both memory and time. On Tue, Oct 8, 2013 at 10:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix

### Re: [Numpy-discussion] Efficient square distance computation

Out of interest, how did you do this with matrix multiplication? http://stackoverflow.com/a/4856692 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

### Re: [Numpy-discussion] Efficient square distance computation

On Tue, Oct 8, 2013 at 1:38 PM, Ke Sun sunk...@gmail.com wrote: On a machine I had access to it took about 20 minutes. How? I am using matrix multiplication (the same code as http://stackoverflow.com/a/4856692) and it runs for around 18 hours. make sure you are using an optimized BLAS

### Re: [Numpy-discussion] Segfault with QR Decomposition

yes thats probably openblas fault. Openblas crashes all the time as soon as your matrices get bigger than a couple of megabytes. I'll investigate and report it upstream (as I have already far too often for the exact same reason ...) On Wed, Oct 9, 2013 at 5:05 PM, Charanpal Dhanjal

### Re: [Numpy-discussion] Segfault with QR Decomposition

https://github.com/xianyi/OpenBLAS/issues/304 On 09.10.2013 17:24, Julian Taylor wrote: yes thats probably openblas fault. Openblas crashes all the time as soon as your matrices get bigger than a couple of megabytes. I'll investigate and report it upstream (as I have already far too often

### Re: [Numpy-discussion] Reading from binary file with memmap, with offset

you have to use the offset keyword argument of np.memmap, else it will always start from the beginning of the file np.memmap(fd, dtype=float32, mode=r, offset=offset) On Thu, Oct 10, 2013 at 2:43 PM, Andreas Hilboll li...@hilboll.de wrote: Hi, I have a problem using memmap correctly. I

### Re: [Numpy-discussion] Bug in numpy.correlate documentation

On 10.10.2013 21:31, Bernhard Spinnler wrote: On 10.10.2013, at 19:27, David Goldsmith d.l.goldsm...@gmail.com mailto:d.l.goldsm...@gmail.com wrote: On Wed, Oct 9, 2013 at 7:48 PM, Bernhard Spinnler bernhard.spinn...@gmx.net mailto:bernhard.spinn...@gmx.net wrote: Hi Richard,

### Re: [Numpy-discussion] 1.7.2

On 29.10.2013 21:00, Charles R Harris wrote: On Tue, Oct 29, 2013 at 1:57 PM, Charles R Harris charlesr.har...@gmail.com mailto:charlesr.har...@gmail.com wrote: Hi All, I'm going to tag 1.7.2 soon. That is, unless someone else would like the experience of making a

### [Numpy-discussion] ANN: NumPy 1.7.2rc1 release

will follow soon. Concerning OS X currently only a single person can create the binary installers which is not a good situation. If you have a suitable machine [0] and want to help out please contact us. Cheers, Julian Taylor [0] we currently base the releases on macos 10.6 using python.org python

### Re: [Numpy-discussion] Caution about using intrisincs, and other 'advanced' optimizations

On 12.11.2013 03:17, David Cournapeau wrote: Hi there, I have noticed more and more subtle and hard to track serious bugs in numpy and scipy, due to the use of advanced optimization features (flags, or gcc intrinsics). I am wondering whether those are worth it: they compile wrongly under

### Re: [Numpy-discussion] Caution about using intrisincs, and other 'advanced' optimizations

On 13.11.2013 18:26, David Cournapeau wrote: Can you narrow it down to a specific intrinsic? they can be enabled and disabled in set ./numpy/core/setup_common.py valgrind shows quite a few invalid read in BOOL_ functions when running the scipy or sklearn test suite.

### Re: [Numpy-discussion] Caution about using intrisincs, and other 'advanced' optimizations

Will do, but the errors I am seeing only appear in the simc.inc.src-based implementation of BOOL_logical_or (they disappear if I disable the simd intrinsics manually in the numpy headers). that is because the simd code always looks at the stride (as it only can run with unit strides) while

### Re: [Numpy-discussion] numpy vbench-marking, compiler comparison

On 25.11.2013 02:32, Yaroslav Halchenko wrote: On Tue, 15 Oct 2013, Nathaniel Smith wrote: What do you have to lose? btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ . I have tuned benchmarking so it now reflects the best performance across multiple executions of

### Re: [Numpy-discussion] numpy vbench-marking, compiler comparison

there isn't that much code in numpy that profits from modern x86 instruction sets, even the simple arithmetic loops are strided and thus unvectorizable by the compiler. They have been vectorized manually in 1.8 using sse2 and it is on my todo list to add runtime detected avx support. On

### Re: [Numpy-discussion] numpy datetime64 NaT string conversion bug patch

On 27.11.2013 21:51, Charles G. Waldman wrote: If you convert an array of strings to datetime64s and 'NaT' (or one of its variants) appears in the string, all subsequent values are rendered as NaT: thanks, a little embarrassing I didn't spot that when I fixed a different bug in the function

### Re: [Numpy-discussion] -ffast-math

On 29.11.2013 21:15, Dan Goodman wrote: Hi, Is it possible to get access to versions of ufuncs like sin and cos but compiled with the -ffast-math compiler switch? I recently noticed that my weave.inline code was much faster for some fairly simple operations than my pure numpy code, and

### Re: [Numpy-discussion] -ffast-math

On 01.12.2013 21:53, Dan Goodman wrote: Julian Taylor jtaylor.debian at googlemail.com writes: can you show the code that is slow in numpy? which version of gcc and libc are you using? with gcc 4.8 it uses the glibc 2.17 sin/cos with fast-math, so there should be no difference. In trying

### Re: [Numpy-discussion] -ffast-math

On 01.12.2013 22:59, Dan Goodman wrote: Julian Taylor jtaylor.debian at googlemail.com writes: your sin and exp calls are loop invariants, they do not depend on the loop iterable. This allows to move the expensive functions out of the loop and only leave some simple arithmetic in the body

### Re: [Numpy-discussion] math.fsum() like ufunc for numpy

related this PR attempts to improve the accuracy of summation: https://github.com/numpy/numpy/pull/3685 but math.fsum gives the exact result so it would a valuable ufunc even when that PR is merged. python3.4 will have yet another accurate summation in the statistics module:

### Re: [Numpy-discussion] nasty bug in 1.8.0??

I opened a ticket for it, though thinking about it, its probably intentional be intentional to find code that assumes it can use the strides to get the itemsize. https://github.com/numpy/numpy/issues/4091 On 02.12.2013 20:35, Neal Becker wrote: I don't think that behavior is acceptable.

### Re: [Numpy-discussion] distributing wheels SSE/superpack options

On 06.12.2013 19:06, Ralf Gommers wrote: Hi all, There are a few discussions on packaging for the scientific Python stack ongoing, on the NumFOCUS and distutils lists: https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg https://groups.google.com/forum/#!topic/numfocus/HUcwXTM_jNY

### Re: [Numpy-discussion] building 32 bit numpy on 64 bit linux

On 12.12.2013 19:58, David Jones wrote: I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message: Broken toolchain: cannot link a simple C program It get's the compile flags right, but not the linker: C compiler: gcc -pthread -fno-strict-aliasing -g

### Re: [Numpy-discussion] building 32 bit numpy on 64 bit linux

On 12.12.2013 20:40, David Jones wrote: On 12/12/13 15:54, Julian Taylor wrote: On 12.12.2013 19:58, David Jones wrote: I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message: Broken toolchain: cannot link a simple C program ... this might work: CC

### Re: [Numpy-discussion] building 32 bit numpy on 64 bit linux

On 13.12.2013 18:46, David Jones wrote: ... Correction. Of course LD_LIBRARY_PATH isn't seen by the compiler. It only applies at run time. How embarrasing:) This isn't the first time I've been bitten by that. I don't mind doing that with manual builds, but what about with pip? Is

### Re: [Numpy-discussion] Need help with np.ma.median and np.apply_along_axis

On 27.12.2013 10:54, alex wrote: median is faster in version 1.8 ___ unfortunately that won't help here because masked median uses apply_along_axis again which is very slow, especially if one wants to calculate thousands of medians of 7 elements as in

### Re: [Numpy-discussion] Loading large NIfTI file - MemoryError

On 31.12.2013 14:13, Amira Chekir wrote: Hello together, I try to load a (large) NIfTI file (DMRI from Human Connectome Project, about 1 GB) with NiBabel. import nibabel as nib img = nib.load(dmri.nii.gz) data = img.get_data() The program crashes during img.get_data() with an

### [Numpy-discussion] ANN: NumPy 1.7.2 release

to the last release candidate four additional minor issues have been fixed and compatibility with python 3.4b1 improved. Source tarballs, installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.2 Cheers, Julian Taylor

### Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 88, Issue 1

On 01.01.2014 16:50, Amira Chekir wrote: On 31.12.2013 14:13, Amira Chekir wrote: Hello together, I try to load a (large) NIfTI file (DMRI from Human Connectome Project, about 1 GB) with NiBabel. import nibabel as nib img = nib.load(dmri.nii.gz) data = img.get_data() The

### Re: [Numpy-discussion] Speedup by avoiding memory alloc twice in scalar array

On 18.07.2013 15:36, Nathaniel Smith wrote: On Wed, Jul 17, 2013 at 5:57 PM, Frédéric Bastien no...@nouiz.org wrote: On Wed, Jul 17, 2013 at 10:39 AM, Nathaniel Smith n...@pobox.com wrote: On Tue, Jul 16, 2013 at 11:55 AM, Nathaniel Smith n...@pobox.com wrote: It's entirely possible I

### Re: [Numpy-discussion] adding fused multiply and add to numpy

On Thu, Jan 9, 2014 at 3:54 PM, Daπid davidmen...@gmail.com wrote: On 8 January 2014 22:39, Julian Taylor jtaylor.deb...@googlemail.comwrote: As you can see even without real hardware support it is about 30% faster than inplace unblocked numpy due better use of memory bandwidth. Its even

### Re: [Numpy-discussion] adding fused multiply and add to numpy

) or at least kahan summation, full overflow/invalid checks etc On Thu, Jan 9, 2014 at 9:43 AM, Freddie Witherden fred...@witherden.org wrote: On 08/01/14 21:39, Julian Taylor wrote: An issue is software emulation of real fma. This can be enabled in the test ufunc with npfma.set_type(libc

### Re: [Numpy-discussion] adding fused multiply and add to numpy

On 10.01.2014 01:49, Frédéric Bastien wrote: Do you know if those instruction are automatically used by gcc if we use the good architecture parameter? they are used if you enable -ffp-contract=fast. Do not set it to `on` this is an alias to `off` due to the semantics of C. -ffast-math

### Re: [Numpy-discussion] Memory allocation cleanup

On Fri, Jan 10, 2014 at 3:48 AM, Nathaniel Smith n...@pobox.com wrote: On Thu, Jan 9, 2014 at 11:21 PM, Charles R Harris charlesr.har...@gmail.com wrote: [...] After a bit more research, some further points to keep in mind: Currently, PyDimMem_* and PyArray_* are just aliases for

### Re: [Numpy-discussion] Memory allocation cleanup

On 10.01.2014 17:03, Nathaniel Smith wrote: On Fri, Jan 10, 2014 at 9:18 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On Fri, Jan 10, 2014 at 3:48 AM, Nathaniel Smith n...@pobox.com wrote: [...] For this reason and missing calloc I don't think we should use the Python API for data

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

On 01/15/2014 11:25 AM, Daπid wrote: On 15 January 2014 11:12, Hedieh Ebrahimi hedieh.ebrah...@amphos21.com mailto:hedieh.ebrah...@amphos21.com wrote: I try to print my fileContent array after I read it and it looks like this :

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

On 01/15/2014 01:38 PM, Julian Taylor wrote: On 01/15/2014 11:25 AM, Daπid wrote: On 15 January 2014 11:12, Hedieh Ebrahimi hedieh.ebrah...@amphos21.com ... for utf 8 data: d = np.loadtxt(file, dtype='utf8') ups this is a very bad example as we can't have utf8 as its variable length

### Re: [Numpy-discussion] adding more unicode dtypes

On 15.01.2014 18:57, Charles R Harris wrote: ... There was a discussion of this long ago and UCS-4 was chosen as the numpy standard. There are just too many complications that arise in supporting both. my guess is that that discussion was before python3 and you could still simply treat

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

On 16.01.2014 00:42, Chris Barker wrote: bump back to the OP: On Wed, Jan 15, 2014 at 2:12 AM, Hedieh Ebrahimi hedieh.ebrah...@amphos21.com mailto:hedieh.ebrah...@amphos21.com wrote: fileContent=loadtxt(filePath,dtype=str) do either of these work for you?

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

This thread is getting a little out of hand which is my fault for initially mixing different topics in one mail, so let me try to summarize: We have three issues here: - a loadtxt bug when loading strings in python3 this has nothing to do with encodings or dtypes it is a bug that should be fixed.

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

On Fri, Jan 17, 2014 at 1:44 PM, Oscar Benjamin oscar.j.benja...@gmail.comwrote: On Fri, Jan 17, 2014 at 10:59:27AM +, Pauli Virtanen wrote: Julian Taylor jtaylor.debian at googlemail.com writes: [clip] For backward compatibility we *cannot* change S. Do you mean to say

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

On Fri, Jan 17, 2014 at 2:10 PM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On Fri, Jan 17, 2014 at 1:44 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote:... ... No latin1 de/encoding is required for anything, I don't know why you would want do to that in this context. Does

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

On Fri, Jan 17, 2014 at 2:40 PM, Oscar Benjamin oscar.j.benja...@gmail.comwrote: On Fri, Jan 17, 2014 at 02:10:19PM +0100, Julian Taylor wrote: On Fri, Jan 17, 2014 at 1:44 PM, Oscar Benjamin oscar.j.benja...@gmail.comwrote: On Fri, Jan 17, 2014 at 10:59:27AM +, Pauli Virtanen wrote

### Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

On 17.01.2014 15:12, Julian Taylor wrote: On Fri, Jan 17, 2014 at 2:40 PM, Oscar Benjamin oscar.j.benja...@gmail.com mailto:oscar.j.benja...@gmail.com wrote: On Fri, Jan 17, 2014 at 02:10:19PM +0100, Julian Taylor wrote: On Fri, Jan 17, 2014 at 1:44 PM, Oscar Benjamin

### Re: [Numpy-discussion] accumulation operation

On 22.01.2014 18:23, Ralf Juengling wrote: Executing the following code, import numpy as np a = np.zeros((3,)) w = np.array([0, 1, 0, 1, 2]) v = np.array([10.0, 1, 10.0, 2, 9]) a[w] += v I was expecting ‘a’ to be array([20., 3., 9.]. Instead I get a array([

### Re: [Numpy-discussion] MKL and OpenBLAS

On 26.01.2014 18:06, Stéfan van der Walt wrote: On Sun, 26 Jan 2014 16:40:44 +0200, Pauli Virtanen wrote: The Numpy Windows binaries distributed in the numpy project at sourceforge.net are compiled with ATLAS, which should count as an optimized BLAS. I don't recall what's the situation with

### Re: [Numpy-discussion] MKL and OpenBLAS

On 26.01.2014 22:33, Sturla Molden wrote: Julian Taylor jtaylor.deb...@googlemail.com wrote: if this issue disqualifies accelerate, it also disqualifies openblas as a default. openblas has the same issue, we stuck a big fat warning into the docs (site.cfg) for this now as people keep running

### [Numpy-discussion] windows and C99 math

hi, numpys no-C99 fallback keeps turning up issues in corner cases, e.g. hypot https://github.com/numpy/numpy/issues/2385 log1p https://github.com/numpy/numpy/issues/4225 these only seem to happen on windows, on linux and mac it seems to use the C99 math library just fine. Are our binary builds

### Re: [Numpy-discussion] scipy image processing memory leak in python 2.7?

On 28.01.2014 19:44, Joseph McGlinchy wrote: Hi numpy list! I am trying to do some image processing on a number of images, 72 to be specific. I am seeing the python memory usage continually increase. which version of scipy are you using? there is a memory leak in ndimage.label in

### Re: [Numpy-discussion] Memory leak in numpy?

On 29.01.2014 20:44, Nathaniel Smith wrote: On Wed, Jan 29, 2014 at 7:39 PM, Joseph McGlinchy jmcglin...@esri.com wrote: Upon further investigation, I do believe it is within the scipy code where there is a leak. I commented out my call to processBinaryImage(), which is all scipy code calls,

### Re: [Numpy-discussion] Memory leak?

which version of numpy are you using? there seems to be a leak in the scalar return due to the PyObject_Malloc usage in git master, but it doesn't affect 1.8.0 On Fri, Jan 31, 2014 at 7:20 AM, Chris Laumann chris.laum...@gmail.comwrote: Hi all- The following snippet appears to leak memory

### Re: [Numpy-discussion] Memory leak?

On 31.01.2014 18:12, Nathaniel Smith wrote: On Fri, Jan 31, 2014 at 4:29 PM, Benjamin Root ben.r...@ou.edu wrote: Just to chime in here about the SciPy Superpack... this distribution tracks the master branch of many projects, and then puts out releases, on the assumption that master contains

### Re: [Numpy-discussion] striding through arbitrarily large files

On Tue, Feb 4, 2014 at 4:27 PM, RayS r...@blue-cove.com wrote: At 07:09 AM 2/4/2014, you wrote: On 04/02/2014 16:01, RayS wrote: I was struggling with methods of reading large disk files into numpy efficiently (not FITS or .npy, just raw files of IEEE floats from numpy.tostring()).

### Re: [Numpy-discussion] Memory leak?

Sent with Airmail On January 31, 2014 at 9:31:40 AM, Julian Taylor (jtaylor.deb...@googlemail.com mailto://jtaylor.deb...@googlemail.com) wrote: On 31.01.2014 18:12, Nathaniel Smith wrote: On Fri, Jan 31, 2014 at 4:29 PM, Benjamin Root ben.r...@ou.edu wrote: Just to chime in here about

### Re: [Numpy-discussion] MKL and OpenBLAS

On Thu, Feb 6, 2014 at 1:11 PM, Thomas Unterthiner thomas_unterthi...@web.de wrote: On 2014-02-06 11:10, Sturla Molden wrote: BTW: The performance of OpenBLAS is far behind Eigen, MKL and ACML, but better than ATLAS and Accelerate. Hi there! Sorry for going a bit off-topic, but: do you

### Re: [Numpy-discussion] Geometrically defined masking arrays; how to optimize?

meshgrid also has the sparse keyword argument which archives the same. On 12.02.2014 20:04, Chris Barker wrote: An extra note here: One of the great things about numpy (as opposed, to say, MATLAB), is array broadcasting Thus you generally don't need meshgrid -- why carry all that extra

### Re: [Numpy-discussion] argsort speed

On 17.02.2014 15:18, Francesc Alted wrote: On 2/17/14, 1:08 AM, josef.p...@gmail.com wrote: On Sun, Feb 16, 2014 at 6:12 PM, Daπid davidmen...@gmail.com wrote: On 16 February 2014 23:43, josef.p...@gmail.com wrote: What's the fastest argsort for a 1d array with around 28 Million elements,

### [Numpy-discussion] allocated memory cache for numpy

hi, I noticed that during some simplistic benchmarks (e.g. https://github.com/numpy/numpy/issues/4310) a lot of time is spent in the kernel zeroing pages. This is because under linux glibc will always allocate large memory blocks with mmap. As these pages can come from other processes the kernel

### Re: [Numpy-discussion] allocated memory cache for numpy

On 17.02.2014 21:16, Sturla Molden wrote: Julian Taylor jtaylor.deb...@googlemail.com wrote: When an array is created it tries to get its memory from the cache and when its deallocated it returns it to the cache. Good idea, however there is already a C function that does this. It uses

### Re: [Numpy-discussion] allocated memory cache for numpy

On 17.02.2014 22:27, Sturla Molden wrote: Nathaniel Smith n...@pobox.com wrote: Also, I'd be pretty wary of caching large chunks of unused memory. People already have a lot of trouble understanding their program's memory usage, and getting rid of 'greedy free' will make this even worse. A

### Re: [Numpy-discussion] allocated memory cache for numpy

On Tue, Feb 18, 2014 at 1:47 AM, David Cournapeau courn...@gmail.com wrote: On Mon, Feb 17, 2014 at 7:31 PM, Julian Taylor jtaylor.deb...@googlemail.com wrote: hi, I noticed that during some simplistic benchmarks (e.g. https://github.com/numpy/numpy/issues/4310) a lot of time is spent

### Re: [Numpy-discussion] allocated memory cache for numpy

On Mon, Feb 17, 2014 at 9:42 PM, Nathaniel Smith n...@pobox.com wrote: On 17 Feb 2014 15:17, Sturla Molden sturla.mol...@gmail.com wrote: Julian Taylor jtaylor.deb...@googlemail.com wrote: When an array is created it tries to get its memory from the cache and when its deallocated

### Re: [Numpy-discussion] allocated memory cache for numpy

On 18.02.2014 16:21, Julian Taylor wrote: On Mon, Feb 17, 2014 at 9:42 PM, Nathaniel Smith n...@pobox.com wrote: On 17 Feb 2014 15:17, Sturla Molden sturla.mol...@gmail.com wrote: Julian Taylor jtaylor.deb...@googlemail.com wrote: When an array is created it tries to get its memory from

### Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On Thu, Feb 20, 2014 at 1:25 AM, Nathaniel Smith n...@pobox.com wrote: Hey all, Just a heads up: thanks to the tireless work of Olivier Grisel, the OpenBLAS development branch is now fork-safe when built with its default threading support. (It is still not thread-safe when built using OMP for

### Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On Thu, Feb 20, 2014 at 3:50 PM, Olivier Grisel olivier.gri...@ensta.org wrote: Thanks for sharing, this is all very interesting. Have you tried to have a look at the memory usage and import time of numpy when linked against libopenblas.dll? -- this is probably caused by the memory warmup

### Re: [Numpy-discussion] cPickle.loads and Numeric

On Tue, Feb 25, 2014 at 5:41 PM, Alexander Belopolsky ndar...@mac.com wrote: On Tue, Feb 25, 2014 at 11:29 AM, Benjamin Root ben.r...@ou.edu wrote: I seem to recall reading somewhere that pickles are not intended to be long-term archives as there is no guarantee that a pickle made in one

### Re: [Numpy-discussion] Help Understanding Indexing Behavior

On 26.02.2014 00:04, JB wrote: At the risk of igniting a flame war...can someone please help me understand the indexing behavior of NumPy? I will readily I admit I come from a Matlab background, but I appreciate the power of Python and am trying to learn more. From a Matlab user's

### [Numpy-discussion] last call for fixes for numpy 1.8.1rc1

hi, We want to start preparing the release candidate for the bugfix release 1.8.1rc1 this weekend, I'll start preparing the changelog tomorrow. So if you want a certain issue fixed please scream now or better create a pull request/patch on the maintenance/1.8.x branch. Please only consider

### Re: [Numpy-discussion] last call for fixes for numpy 1.8.1rc1

while. This issue limits quite a bit the number of operands in numexpr expressions, and hence, to other projects that depends on it, like PyTables or pandas. See for example this bug report: https://github.com/PyTables/PyTables/issues/286 Thanks, Francesc On 2/27/14, 9:05 PM, Julian

### Re: [Numpy-discussion] last call for fixes for numpy 1.8.1rc1

, increasing the temporary arrays in nditer from 32kb to 128kb is a bit worrying, but probably we should do some benchmarks and see how much performance would be compromised (if any). Francesc On 2/28/14, 1:09 PM, Julian Taylor wrote: hm increasing it for PyArrayMapIterObject would break the public

### Re: [Numpy-discussion] 2D array indexing

On 01.03.2014 00:32, Gökhan Sever wrote: Hello, Given this simple 2D array: In [1]: np.arange(9).reshape((3,3)) Out[1]: array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) In [2]: a = np.arange(9).reshape((3,3)) In [3]: a[:1:] Out[3]: array([[0, 1, 2]]) In [4]: a[:1,:]

### [Numpy-discussion] numpy gsoc topic idea: configurable algorithm precision and vector math library integration

hi, as the numpy gsoc topic page is a little short on options I was thinking about adding two topics for interested students. But as I have no experience with gsoc or mentoring and the ideas are not very fleshed out yet I'd like to ask if it might make sense at all: 1. configurable algorithm

### Re: [Numpy-discussion] 1.8.1rc1 on sourceforge.

On 04.03.2014 18:08, Christoph Gohlke wrote: On 3/4/2014 4:49 AM, Thomas Unterthiner wrote: Hi there! I just tried setting up a new installation using numpy 1.8.1rc1 (+scipy 0.13.3 and matplotlib 1.3.1). I ran into problems when installing matplotlib 1.3.1. The attached logfile shows the

### Re: [Numpy-discussion] 1.8.1 release

On 06.03.2014 19:46, Skipper Seabold wrote: Hi, Should [1] be considered a release blocker for 1.8.1? Skipper [1] https://github.com/numpy/numpy/issues/4442 as far as I can tell its a regression of the 1.8.0 release but not the 1.8.1 release so I wouldn't consider it a blocker. But its

### Re: [Numpy-discussion] 1.8.1rc1 on sourceforge.

this was caused by something in the rc. Cheers Thomas On 2014-03-03 17:23, Charles R Harris wrote: Hi All, Julian Taylor has put windows binaries and sources for the 1.8.1 release candidate up on sourceforge http://sourceforge.net/projects/numpy/files/NumPy/1.8.1rc1/. If things go well

### Re: [Numpy-discussion] GSoC project: draft of proposal

libraries provide can be very useful for this. Regards, Julian Taylor ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

### Re: [Numpy-discussion] Drop support for Python 3.2?

On 25.03.2014 00:28, Charles R Harris wrote: Hi All, The suggestion has been made the we drop Python 3.2 support in numpy 1.9 and scipy 0.15. The advantage, from my point of view, to supporting Python = 3.3 is that the u'unicode' syntax is supported in 3.3 and this makes it easier to

### [Numpy-discussion] ANN: NumPy 1.8.1 release

party applications. Please check the release notes for details. Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.1 Cheers, Julian Taylor signature.asc Description: OpenPGP digital signature

### Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On 26.03.2014 16:27, Olivier Grisel wrote: Hi Carl, I installed Python 2.7.6 64 bits on a windows server instance from rackspace cloud and then ran get-pip.py and then could successfully install the numpy and scipy wheel packages from your google drive folder. I tested dot products and

### Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On 26.03.2014 21:41, Nathaniel Smith wrote: On Wed, Mar 26, 2014 at 7:34 PM, Julian Taylor jtaylor.deb...@googlemail.com wrote: as for using openblas by default in binary builds, no. pthread openblas build is now fork safe which is great but it is still not reliable enough for a default. E.g