Paulo van Breugel wrote: > >> With memmap you still have a limits of 2Gb I guess, you should try: dask > >> > > > Just reading the memmap manual page, where it reads: "Memory-mapped arrays > use the Python memory-map object which (prior to Python 2.5) does not allow > files to be larger than a certain size depending on the platform. This size > is always < 2GB even on 64-bit systems.". Which is unclear to me; I am not > sure if that means that this limit is different or does not apply when on > Python 2.5 or newer (what is the minimum python version for GRASS?)
Even if Python itself no longer imposes a 2 GiB limit, you would probably need to be using a 64-bit platform, the system would need enough memory for the operation, and it would need to allow the process to use that much memory. The problem here is that mahdist = np.sum(np.sum(delta[None,:,:,:] * VI[:,:,None,None],axis=1) * delta,axis=0) expands both delta and VI to 4-D arrays then calculates their product element-wise, then sums them over two axes. The intermediate array of products could potentially be very large. It may be possible to avoid this issue using numpy.tensordot, e.g. mahdist = np.sum(np.tensordot(VI, delta, (1,0)) * delta,axis=0) I don't know for sure whether this actually uses less memory. It could do, as it rearranges the matrices so that the sum-of-products is calculated using np.dot, which is a built-in function. -- Glynn Clements <[email protected]> _______________________________________________ grass-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-dev
