The Gramian is 8000 x 8000, dense, and full of 8-byte doubles. It's
symmetric so can get away with storing it in ~256MB. The catch is that
it's going to send around copies of this 256MB array. You may easily
be running your driver out of memory given all the overheads and
copies, or your executors,
I got this error when trying to perform PCA on a sparse matrix, each row
has a nominal length of 8000, and there are 36k rows. each row has on
average 3 elements being non-zero.
I guess the total size is not that big.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.