Dear all, Thank you very much for your help and suggestions! I have learned a lot from you. Indeed, I want to use this matrix and not only to read and store the data.
I should have written in my email that I was wondering if Julia had something similar to bigmemory and bigalgebra in R ( https://sites.google.com/site/bigmemoryorg/home/bigalgebra). I know they make the allocation of huge amounts of data easier, but I didn't try to do it for my data yet (all my project is in Julia, so I want to try it until the infinite before using other languages). As it seems to be a problem related to my computational resources, my first shot will be to try to run it in Julia in a cluster with more memory. If it doesn't work I will take some time to think in a different way to solve my problem (in worst case I will need to avoid this high-resolution dataset for now). Thanks a lot! Charles On 14 August 2015 at 00:18, Marcio Sales <[email protected]> wrote: > Charles, > If your work ends with the matrix, then you could write a binary file on > dink, and make your loop calculate and hold a bunch of values each time and > save it to the binary file on disk periodically, until all values are > saved. Try to make the loop hold as many values as possible, so you don't > need to save as frequently, to improve performance. > > But If you then need to use this matrix for other operations, then you > have a problem... unless you buy enough memory, or you can somehow do the > operation in parts. > Marcio > > > Em quinta-feira, 13 de agosto de 2015 07:26:55 UTC-3, Charles Santana > escreveu: > >> Hi all, >> >> Do you recommend a way to work with bit matrices in Julia. By "big" I >> mean a 65600 x 65600 symmetric matrix (the upper triangular matrix is equal >> to the lower triangular one). >> >> I am studying the distances between sites at different resolution maps. >> For low resolution we have few sites, and for big resolution we have more >> sites (S). >> >> For few sites (small matrices) I was doing something like this: >> >> S = 100;#number of sites >> M = zeros(S,S); >> for i in 1:(S-1) >> for j in (i+1):S >> M[i,j] = dist(i,j);#where dist(i,j) is the distance between sites i >> and j >> end >> end >> >> However, for big matrices I get the following message: >> >> S=65600; >> M = zeros(S,S); >> ERROR: OutOfMemoryError() >> in call at essentials.jl:201 >> in zeros at array.jl:233 >> >> I am using Julia Version 0.4.0-dev+5920 in Ubuntu 14.04. >> >> Thanks for any tip! >> >> Best, >> >> Charles >> -- >> Um axé! :) >> >> -- >> Charles Novaes de Santana, PhD >> http://www.imedea.uib-csic.es/~charles >> > -- Um axé! :) -- Charles Novaes de Santana, PhD http://www.imedea.uib-csic.es/~charles
