Re: DoubleMatrix vs Array[Array[Double]] : Question about debugging serialization performance issues

[Guillaume Pitel]

Hi Mattei, Memory-wise, the dense matrices are quite small actually : depending on my chuncking, it can be between 15MB in memory (50000 rows, 40 cols, 8bytes/double) up to 60MB for my craziest attempts :) Each chunk contains two such matrices (one IN and one OUT) during the work, each partition may typically deal with  50 chunks Guillaume Hi Guillaume, I haven’t looked at the serialization of DoubleMatrix but I believe it just creates one big Array[Double] instead of many ones, and stores all the rows contiguously in that. I don’t think that would be slower to serialize. However, because the object is bigger overall, it might need to get allocated in another part of the heap (e.g. instead of in the new generation), which causes more GC and may cause out-of-memory sooner. How big are these matrices? You might want to calculate what exactly is taking up memory. Matei -- Guillaume PITEL, Président +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53 eXenSa S.A.S. 41, rue Périer - 92120 Montrouge - FRANCE Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05