Hi List, I had similar problems on windows. I tried to use memmaps to buffer a large amount of data and process it in chunks. But I found that whenever I tried to do this, I always ended filling up RAM completely which led to crashes of my python script with a MemoryError. This led me to consider, actually from an advice via this list, the module h5py, which has a nice numpy interface to the hdf5 file format. It seemed more clear to me with the h5py-module, what was being buffered on disk and what was stored in RAM.
Cheers, Simon On Sun, Oct 24, 2010 at 2:15 AM, David Cournapeau <[email protected]>wrote: > On Sun, Oct 24, 2010 at 12:44 AM, braingateway <[email protected]> > wrote: > > > > > I agree with you about the point of using memmap. > > > That is why the behavior > > is so strange to me. > > I think it is expected. What kind of behavior were you expecting ? To > be clear, if I have a lot of available ram, I expect memmap arrays to > take almost all of it (virtual memroy ~ resident memory). Now, if at > the same time, another process starts taking a lot of memory, I expect > the OS to automatically lower resident memory for the process using > memmap. > > I did a small experiment on mac os x, creating a giant mmap'd array in > numpy, and at the same time running a small C program using mlock (to > lock pages into physical memory). As soon as I lock a big area (where > big means most of my physical ram), the python process dealing with > the mmap area sees its resident memory decrease. As soon as I kill the > C program locking the memory, the resident memory starts increasing > again. > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > [email protected] > http://mail.scipy.org/mailman/listinfo/numpy-discussion >
_______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
