Re: [Numpy-discussion] np.memmap and memory usage

2009-07-01 Thread Pauli Virtanen
Wed, 01 Jul 2009 10:17:51 +0200, Emmanuelle Gouillart kirjoitti: I'm using numpy.memmap to open big 3-D arrays of Xray tomography data. After I have created a new array using memmap, I modify the contrast of every Z-slice (along the first dimension) inside a for loop, for a better

Re: [Numpy-discussion] np.memmap and memory usage

2009-07-01 Thread Francesc Alted
A Wednesday 01 July 2009 10:17:51 Emmanuelle Gouillart escrigué: Hello, I'm using numpy.memmap to open big 3-D arrays of Xray tomography data. After I have created a new array using memmap, I modify the contrast of every Z-slice (along the first dimension) inside a for loop, for

Re: [Numpy-discussion] np.memmap and memory usage

2009-07-01 Thread Emmanuelle Gouillart
Hi Pauli, thank you for your answer! I was indeed measuring the memory used with top, which is not the best tool for understanding what really happens. I monitored free during the execution of my program and indeed, the used numbers on the +/-buffers/cache line stays roughly

Re: [Numpy-discussion] np.memmap and memory usage

2009-07-01 Thread Emmanuelle Gouillart
Hi Francesc, many thanks for this very detailed and informative answer! This list is really great :D. I'm going to install pytables at once and I will try your scripts with my data. As your computations were made with files of the same size as mine, hopefully it should

Re: [Numpy-discussion] np.memmap and memory usage

2009-07-01 Thread David Cournapeau
On Wed, Jul 1, 2009 at 6:14 PM, Pauli Virtanenp...@iki.fi wrote: Wed, 01 Jul 2009 10:17:51 +0200, Emmanuelle Gouillart kirjoitti:       I'm using numpy.memmap to open big 3-D arrays of Xray tomography data. After I have created a new array using memmap, I modify the contrast of every Z-slice

Re: [Numpy-discussion] np.memmap and memory usage

2009-07-01 Thread Francesc Alted
A Wednesday 01 July 2009 15:04:08 Francesc Alted escrigué: However, you can still speed-up out-of-core computations by using the recently introduced tables.Expr class (PyTables 2.2b1, see [2]), which uses a combination of the Numexpr [3] and PyTables advanced computing capabilities: