With 24GB RAM, 12G for the input, and almost the same again for the sparse matrix generated will certainly run it dangerously close to the total memory.
-viral On Tuesday, October 7, 2014 9:30:27 PM UTC+5:30, Iain Dunning wrote: > > Yes I think your estimate is correct: > > N => 10001 > k => 500080001 > num_bytes => 12001920024 > num_bytes / 1024^3 => 11.177659057080746 > > When I run the code, it doesn't have any trouble allocating the 3 arrays > (I have 16GB ram) and populating them (its pretty quick too), but it > "freezes" when it gets to the `sparse` call. > > I think it ends up here: > > https://github.com/JuliaLang/julia/blob/master/base/sparse/sparsematrix.jl#L238 > and I guess the memory allocated in there is what kills it. > > Maybe open a Julia issue? I don't know how that code works exactly sorry. > > Thanks, > Iain > > > > On Tuesday, October 7, 2014 11:45:24 AM UTC-4, Andrei Berceanu wrote: >> >> Hi Iain, >> >> First of all thanks for your effort! >> I am using Julia 0.3.0 on Linux. The reason i wasn't preallocating was >> because I did not know the array sizes beforehand, but your suggestion of >> computing it works very well :) >> I just noticed that in my original post I put 1000 instead of 10000. So, >> the problem is that for such a large value, Julia crashes. And if I'm not >> mistaken the 3 allocated arrays only occupy around 12GB (out of my 24GB >> RAM). >> >> //A >> >> On Tuesday, October 7, 2014 5:23:11 PM UTC+2, Iain Dunning wrote: >>> >>> OK, on Julia 0.3.0 on OSX, for N=1001 >>> elapsed time: 0.722537543 seconds (960169036 bytes allocated, 11.71% gc >>> time) >>> elapsed time: 0.703546159 seconds (955424928 bytes allocated, 26.01% gc >>> time) >>> elapsed time: 0.692751989 seconds (955424928 bytes allocated, 23.39% gc >>> time) >>> (first run includes JIT) >>> So I'm not sure whats going on for you? >>> >>> Anyway, I tried to improve the speed, again for N=1001 >>> elapsed time: 0.256739122 seconds (321392500 bytes allocated, 23.75% gc >>> time) >>> elapsed time: 0.259208305 seconds (320464768 bytes allocated, 17.49% gc >>> time) >>> elapsed time: 0.212410743 seconds (320464768 bytes allocated, 32.42% gc >>> time) >>> By preallocating everything: >>> https://gist.github.com/IainNZ/c7dd570ffedbf629a81d >>> >>> All the effort is in `sparse` (you can check with profiler) >>> >>> Thanks, >>> Iain >>> >>> On Tuesday, October 7, 2014 11:01:05 AM UTC-4, Iain Dunning wrote: >>>> >>>> At a glance, why build list only to just add it to J? Why not add it >>>> directly to J, and add i to I. >>>> >>>> If I have a chance I'll look further. >>>> >>>> Which Julia? >>>> >>>> On Monday, October 6, 2014 1:51:36 PM UTC-4, Andrei Berceanu wrote: >>>>> >>>>> I have written the following Julia code to build a sparse matrix of >>>>> dimension N^2xN^2 >>>>> >>>>> https://gist.github.com/berceanu/fe7e26840637517383d8 >>>>> >>>>> The code works (probably in a very suboptimal way) for small enough >>>>> matrices, but for example if I set N=1000, genspmat(1000) quickly eats up >>>>> my RAM and crashes Julia. I doubt that this is related to the storage of >>>>> the sparse matrix itself, and suspect it has to do with the garbage >>>>> collection inside the main loop of genspmat, but I have no idea of fixing >>>>> it. Any suggestions? >>>>> >>>>> Thanks! >>>>> >>>>
