If you're careful and know how the compressed sparse column format works, you can try generating the matrix column by column, doing in-place sorting of the row indices within each column. That would make it straightforward to construct the colptr and rowval vectors, and use the SparseMatrixCSC constructor directly without making any extra copies. Take care that you generate a correctly-formed CSC matrix if you go this route though, otherwise some operations on the resulting matrix might not work.
On Thursday, October 9, 2014 10:12:08 AM UTC-7, Viral Shah wrote: > > With 24GB RAM, 12G for the input, and almost the same again for the sparse > matrix generated will certainly run it dangerously close to the total > memory. > > -viral > > On Tuesday, October 7, 2014 9:30:27 PM UTC+5:30, Iain Dunning wrote: >> >> Yes I think your estimate is correct: >> >> N => 10001 >> k => 500080001 >> num_bytes => 12001920024 >> num_bytes / 1024^3 => 11.177659057080746 >> >> When I run the code, it doesn't have any trouble allocating the 3 arrays >> (I have 16GB ram) and populating them (its pretty quick too), but it >> "freezes" when it gets to the `sparse` call. >> >> I think it ends up here: >> >> https://github.com/JuliaLang/julia/blob/master/base/sparse/sparsematrix.jl#L238 >> and I guess the memory allocated in there is what kills it. >> >> Maybe open a Julia issue? I don't know how that code works exactly sorry. >> >> Thanks, >> Iain >> >> >> >> On Tuesday, October 7, 2014 11:45:24 AM UTC-4, Andrei Berceanu wrote: >>> >>> Hi Iain, >>> >>> First of all thanks for your effort! >>> I am using Julia 0.3.0 on Linux. The reason i wasn't preallocating was >>> because I did not know the array sizes beforehand, but your suggestion of >>> computing it works very well :) >>> I just noticed that in my original post I put 1000 instead of 10000. So, >>> the problem is that for such a large value, Julia crashes. And if I'm not >>> mistaken the 3 allocated arrays only occupy around 12GB (out of my 24GB >>> RAM). >>> >>> //A >>> >>> On Tuesday, October 7, 2014 5:23:11 PM UTC+2, Iain Dunning wrote: >>>> >>>> OK, on Julia 0.3.0 on OSX, for N=1001 >>>> elapsed time: 0.722537543 seconds (960169036 bytes allocated, 11.71% gc >>>> time) >>>> elapsed time: 0.703546159 seconds (955424928 bytes allocated, 26.01% gc >>>> time) >>>> elapsed time: 0.692751989 seconds (955424928 bytes allocated, 23.39% gc >>>> time) >>>> (first run includes JIT) >>>> So I'm not sure whats going on for you? >>>> >>>> Anyway, I tried to improve the speed, again for N=1001 >>>> elapsed time: 0.256739122 seconds (321392500 bytes allocated, 23.75% gc >>>> time) >>>> elapsed time: 0.259208305 seconds (320464768 bytes allocated, 17.49% gc >>>> time) >>>> elapsed time: 0.212410743 seconds (320464768 bytes allocated, 32.42% gc >>>> time) >>>> By preallocating everything: >>>> https://gist.github.com/IainNZ/c7dd570ffedbf629a81d >>>> >>>> All the effort is in `sparse` (you can check with profiler) >>>> >>>> Thanks, >>>> Iain >>>> >>>> On Tuesday, October 7, 2014 11:01:05 AM UTC-4, Iain Dunning wrote: >>>>> >>>>> At a glance, why build list only to just add it to J? Why not add it >>>>> directly to J, and add i to I. >>>>> >>>>> If I have a chance I'll look further. >>>>> >>>>> Which Julia? >>>>> >>>>> On Monday, October 6, 2014 1:51:36 PM UTC-4, Andrei Berceanu wrote: >>>>>> >>>>>> I have written the following Julia code to build a sparse matrix of >>>>>> dimension N^2xN^2 >>>>>> >>>>>> https://gist.github.com/berceanu/fe7e26840637517383d8 >>>>>> >>>>>> The code works (probably in a very suboptimal way) for small enough >>>>>> matrices, but for example if I set N=1000, genspmat(1000) quickly eats >>>>>> up >>>>>> my RAM and crashes Julia. I doubt that this is related to the storage of >>>>>> the sparse matrix itself, and suspect it has to do with the garbage >>>>>> collection inside the main loop of genspmat, but I have no idea of >>>>>> fixing >>>>>> it. Any suggestions? >>>>>> >>>>>> Thanks! >>>>>> >>>>>
