On Dec 21, 2007 12:11 AM, Martin Spacek <[EMAIL PROTECTED]> wrote: > >> By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine, > >> and now numpy.memmap works like a charm. Slicing around a 15 GB file is > >> fun! > >> > > Thanks for the feedback ! > > Did you get the kind of speed you need and/or the speed you were hoping for > > ? > > Nope. Like I wrote earlier, it seems there isn't time for disk access in > my main loop, which is what memmap is all about. I resolved this by > loading the whole file into memory as a python list of 2D arrays, > instead of one huge contiguous 3D array. That got me an extra 100 to 200 > MB of physical memory to work with (about 1.4GB out of 2GB total) on > win32, which is all I needed. >
Instead of saying "memmap is ALL about disc access" I would rather like to say that "memap is all about SMART disk access" -- what I mean is that memmap should run as fast as a normal ndarray if it works on the cached part of an array. Maybe there is a way of telling memmap when and what to cache and when to sync that cache to the disk. In other words, memmap should perform just like a in-pysical-memory array -- only that it once-in-a-while saves/load to/from the disk. Or is this just wishful thinking ? Is there a way of "pre loading" a given part into cache (pysical-memory) or prevent disc writes at "bad times" ? How about doing the sync from a different thread ;-) -Sebastian _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
