bigmemory is presumably the same thing as mmap_array. --Tim
On Friday, August 14, 2015 01:03:10 AM Charles Novaes de Santana wrote: > Dear all, > > Thank you very much for your help and suggestions! I have learned a lot > from you. Indeed, I want to use this matrix and not only to read and store > the data. > > I should have written in my email that I was wondering if Julia had > something similar to bigmemory and bigalgebra in R ( > https://sites.google.com/site/bigmemoryorg/home/bigalgebra). I know they > make the allocation of huge amounts of data easier, but I didn't try to do > it for my data yet (all my project is in Julia, so I want to try it until > the infinite before using other languages). > > As it seems to be a problem related to my computational resources, my first > shot will be to try to run it in Julia in a cluster with more memory. If it > doesn't work I will take some time to think in a different way to solve my > problem (in worst case I will need to avoid this high-resolution dataset > for now). > > Thanks a lot! > > Charles > > On 14 August 2015 at 00:18, Marcio Sales <[email protected]> wrote: > > Charles, > > If your work ends with the matrix, then you could write a binary file on > > dink, and make your loop calculate and hold a bunch of values each time > > and > > save it to the binary file on disk periodically, until all values are > > saved. Try to make the loop hold as many values as possible, so you don't > > need to save as frequently, to improve performance. > > > > But If you then need to use this matrix for other operations, then you > > have a problem... unless you buy enough memory, or you can somehow do the > > operation in parts. > > Marcio > > > > > > Em quinta-feira, 13 de agosto de 2015 07:26:55 UTC-3, Charles Santana > > > > escreveu: > >> Hi all, > >> > >> Do you recommend a way to work with bit matrices in Julia. By "big" I > >> mean a 65600 x 65600 symmetric matrix (the upper triangular matrix is > >> equal > >> to the lower triangular one). > >> > >> I am studying the distances between sites at different resolution maps. > >> For low resolution we have few sites, and for big resolution we have more > >> sites (S). > >> > >> For few sites (small matrices) I was doing something like this: > >> > >> S = 100;#number of sites > >> M = zeros(S,S); > >> for i in 1:(S-1) > >> > >> for j in (i+1):S > >> > >> M[i,j] = dist(i,j);#where dist(i,j) is the distance between sites i > >> > >> and j > >> > >> end > >> > >> end > >> > >> However, for big matrices I get the following message: > >> > >> S=65600; > >> M = zeros(S,S); > >> ERROR: OutOfMemoryError() > >> > >> in call at essentials.jl:201 > >> in zeros at array.jl:233 > >> > >> I am using Julia Version 0.4.0-dev+5920 in Ubuntu 14.04. > >> > >> Thanks for any tip! > >> > >> Best, > >> > >> Charles > >> -- > >> Um axé! :) > >> > >> -- > >> Charles Novaes de Santana, PhD > >> http://www.imedea.uib-csic.es/~charles
