Hi Everyone, Thanks for your suggestions and replies. I initially tried what Anne suggested, modifying the strides in the third dimension to account for the 8-byte delimiters between slabs, but I couldn't control the performance as much as I'd like, and I wasn't entirely sure when and where "real" memory was being allocated or deallocated, or how it was striding over the disk. (Memory-mapping also seems to require allocating the full size of the array as virtual memory.) In the end I ended up writing a very short Cython routine hand-tuned for the problem that accepted the specific 3D slab to read off disk and return that to the user, which also helped me coordinate better between multiple MPI processes, as well.
Thanks very much for your suggestions! -Matt On Sun, Apr 25, 2010 at 7:57 AM, Robert Kern <[email protected]> wrote: > On Thu, Apr 22, 2010 at 21:57, David Huard <[email protected]> wrote: >> Hi Matt, >> I don't think the memmap code support this. However, you can stack memmaps >> just as easily as arrays, so if you define individual memmaps for each slice >> and stack them (numpy.vstack), the resulting array will behave as a regular >> 3D array. > > However, that will make a main-memory copy of everything, presumably > what he is trying to avoid. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > [email protected] > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
