Re: RowMatrix PCA out of heap space error

2014-10-13 Thread Sean Owen
The Gramian is 8000 x 8000, dense, and full of 8-byte doubles. It's symmetric so can get away with storing it in ~256MB. The catch is that it's going to send around copies of this 256MB array. You may easily be running your driver out of memory given all the overheads and copies, or your executors,

RowMatrix PCA out of heap space error

2014-10-13 Thread Yang
I got this error when trying to perform PCA on a sparse matrix, each row has a nominal length of 8000, and there are 36k rows. each row has on average 3 elements being non-zero. I guess the total size is not that big. Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.