Yes I think your estimate is correct: N => 10001 k => 500080001 num_bytes => 12001920024 num_bytes / 1024^3 => 11.177659057080746
When I run the code, it doesn't have any trouble allocating the 3 arrays (I have 16GB ram) and populating them (its pretty quick too), but it "freezes" when it gets to the `sparse` call. I think it ends up here: https://github.com/JuliaLang/julia/blob/master/base/sparse/sparsematrix.jl#L238 and I guess the memory allocated in there is what kills it. Maybe open a Julia issue? I don't know how that code works exactly sorry. Thanks, Iain On Tuesday, October 7, 2014 11:45:24 AM UTC-4, Andrei Berceanu wrote: > > Hi Iain, > > First of all thanks for your effort! > I am using Julia 0.3.0 on Linux. The reason i wasn't preallocating was > because I did not know the array sizes beforehand, but your suggestion of > computing it works very well :) > I just noticed that in my original post I put 1000 instead of 10000. So, > the problem is that for such a large value, Julia crashes. And if I'm not > mistaken the 3 allocated arrays only occupy around 12GB (out of my 24GB > RAM). > > //A > > On Tuesday, October 7, 2014 5:23:11 PM UTC+2, Iain Dunning wrote: >> >> OK, on Julia 0.3.0 on OSX, for N=1001 >> elapsed time: 0.722537543 seconds (960169036 bytes allocated, 11.71% gc >> time) >> elapsed time: 0.703546159 seconds (955424928 bytes allocated, 26.01% gc >> time) >> elapsed time: 0.692751989 seconds (955424928 bytes allocated, 23.39% gc >> time) >> (first run includes JIT) >> So I'm not sure whats going on for you? >> >> Anyway, I tried to improve the speed, again for N=1001 >> elapsed time: 0.256739122 seconds (321392500 bytes allocated, 23.75% gc >> time) >> elapsed time: 0.259208305 seconds (320464768 bytes allocated, 17.49% gc >> time) >> elapsed time: 0.212410743 seconds (320464768 bytes allocated, 32.42% gc >> time) >> By preallocating everything: >> https://gist.github.com/IainNZ/c7dd570ffedbf629a81d >> >> All the effort is in `sparse` (you can check with profiler) >> >> Thanks, >> Iain >> >> On Tuesday, October 7, 2014 11:01:05 AM UTC-4, Iain Dunning wrote: >>> >>> At a glance, why build list only to just add it to J? Why not add it >>> directly to J, and add i to I. >>> >>> If I have a chance I'll look further. >>> >>> Which Julia? >>> >>> On Monday, October 6, 2014 1:51:36 PM UTC-4, Andrei Berceanu wrote: >>>> >>>> I have written the following Julia code to build a sparse matrix of >>>> dimension N^2xN^2 >>>> >>>> https://gist.github.com/berceanu/fe7e26840637517383d8 >>>> >>>> The code works (probably in a very suboptimal way) for small enough >>>> matrices, but for example if I set N=1000, genspmat(1000) quickly eats up >>>> my RAM and crashes Julia. I doubt that this is related to the storage of >>>> the sparse matrix itself, and suspect it has to do with the garbage >>>> collection inside the main loop of genspmat, but I have no idea of fixing >>>> it. Any suggestions? >>>> >>>> Thanks! >>>> >>>
