Memory mapped arrays are going to be a performance nightmare if you actually need to write to the entire array. If you want a dense 65600 x 65600 matrix, you need at least 4GB times your element size, which by default is 8 bytes for Float64, which means you need a machine with at least 32GB of RAM, which shouldn't be too hard to find these days. The fact that this is upper triangular can save you half of that, but you'd need to implement custom upper triangular storage for that. The index computations are a bit tricky, but it shouldn't be that bad. If you can figure out some way that the matrix can be made sparse, then you could reduce this considerably, but you'd need to do something clever since most points are not distance zero from each other. The Distances package <https://github.com/JuliaStats/Distances.jl> may be of interest to you as well, but it won't help with the memory issues.
On Thu, Aug 13, 2015 at 10:01 AM, Alex Ames <[email protected]> wrote: > If you're willing to pay the performance cost associated with disk I/O, a > memory-mapped array (mmap_array) might be what you're looking for.
