David Cournapeau :
2010/10/23 braingateway <[email protected]>:Hi everyone, I noticed the numpy.memmap using RAM to buffer data from memmap files. If I get a 100GB array in a memmap file and process it block by block, the RAM usage is going to increasing with the process running until there is no available space in RAM (4GB), even though the block size is only 1MB. for example: #### a = numpy.memmap(‘a.bin’, dtype='float64', mode='r') blocklen=1e5 b=npy.zeros((len(a)/blocklen,)) for i in range(0,len(a)/blocklen): b[i]=npy.mean(a[i*blocklen:(i+1)*blocklen]) #### Is there any way to restrict the memory usage in numpy.memmap?The whole point of using memmap is to let the OS do the buffering for you (which is likely to do a better job than you in many cases). Which OS are you using ? And how do you measure how much memory is taken by numpy for your array ? David _______________________________________________
Hi David,I agree with you about the point of using memmap. That is why the behavior is so strange to me. I actually measure the size of resident set (pink trace in figure2) of the python process on Windows. Here I attached the result. You can see the RAM usage is definitely not file system cache.
LittleBigBrain
<<inline: numpyMemmapAvaRAM.PNG>>
<<inline: numpyMemmapAvaRAM2.PNG>>
_______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
