Maybe you could use a ragged array, which would only store the half of the matrix you need: https://bitbucket.org/maurow/ragged.jl
On Fri, 2015-08-14 at 01:03, Charles Novaes de Santana <[email protected]> wrote: > Dear all, > > Thank you very much for your help and suggestions! I have learned a lot > from you. Indeed, I want to use this matrix and not only to read and store > the data. > > I should have written in my email that I was wondering if Julia had > something similar to bigmemory and bigalgebra in R ( > https://sites.google.com/site/bigmemoryorg/home/bigalgebra). I know they > make the allocation of huge amounts of data easier, but I didn't try to do > it for my data yet (all my project is in Julia, so I want to try it until > the infinite before using other languages). > > As it seems to be a problem related to my computational resources, my first > shot will be to try to run it in Julia in a cluster with more memory. If it > doesn't work I will take some time to think in a different way to solve my > problem (in worst case I will need to avoid this high-resolution dataset > for now). > > Thanks a lot! > > Charles > > On 14 August 2015 at 00:18, Marcio Sales <[email protected]> wrote: > >> Charles, >> If your work ends with the matrix, then you could write a binary file on >> dink, and make your loop calculate and hold a bunch of values each time and >> save it to the binary file on disk periodically, until all values are >> saved. Try to make the loop hold as many values as possible, so you don't >> need to save as frequently, to improve performance. >> >> But If you then need to use this matrix for other operations, then you >> have a problem... unless you buy enough memory, or you can somehow do the >> operation in parts. >> Marcio >> >> >> Em quinta-feira, 13 de agosto de 2015 07:26:55 UTC-3, Charles Santana >> escreveu: >> >>> Hi all, >>> >>> Do you recommend a way to work with bit matrices in Julia. By "big" I >>> mean a 65600 x 65600 symmetric matrix (the upper triangular matrix is equal >>> to the lower triangular one). >>> >>> I am studying the distances between sites at different resolution maps. >>> For low resolution we have few sites, and for big resolution we have more >>> sites (S). >>> >>> For few sites (small matrices) I was doing something like this: >>> >>> S = 100;#number of sites >>> M = zeros(S,S); >>> for i in 1:(S-1) >>> for j in (i+1):S >>> M[i,j] = dist(i,j);#where dist(i,j) is the distance between sites i >>> and j >>> end >>> end >>> >>> However, for big matrices I get the following message: >>> >>> S=65600; >>> M = zeros(S,S); >>> ERROR: OutOfMemoryError() >>> in call at essentials.jl:201 >>> in zeros at array.jl:233 >>> >>> I am using Julia Version 0.4.0-dev+5920 in Ubuntu 14.04. >>> >>> Thanks for any tip! >>> >>> Best, >>> >>> Charles >>> -- >>> Um axé! :) >>> >>> -- >>> Charles Novaes de Santana, PhD >>> http://www.imedea.uib-csic.es/~charles >>> >> > > > -- > Um axé! :)
