This may also be related to https://issues.apache.org/jira/browse/SPARK-22033
On Tue, Sep 19, 2017 at 3:40 PM, Mark Bittmann <mbittm...@gmail.com> wrote: > I've run into this before. The EigenValueDecomposition creates a Java > Array with 2*k*n elements. The Java Array is indexed with a native integer > type, so 2*k*n cannot exceed Integer.MAX_VALUE values. > > The array is created here: > https://github.com/apache/spark/blob/master/mllib/src/main/ > scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala#L84 > > If you remove the requirement that 2*k*n<MAX_VALUE statement, it would > fail with java.lang.NegativeArraySizeException. More here on this issue > here: > https://issues.apache.org/jira/browse/SPARK-5656 > > On Tue, Sep 19, 2017 at 9:49 AM, Alexander Ovcharenko < > shurik....@gmail.com> wrote: > >> Hello guys, >> >> While trying to compute SVD using computeSVD() function, i am getting the >> following warning with the follow up exception: >> 17/09/14 12:29:02 WARN RowMatrix: computing svd with k=49865 and >> n=191077, please check necessity >> IllegalArgumentException: u'requirement failed: k = 49865 and/or n = >> 191077 are too large to compute an eigendecomposition' >> >> When I try to compute first 3000 singular values, I'm getting several >> following warnings every second: >> 17/09/14 13:43:38 WARN TaskSetManager: Stage 4802 contains a task of very >> large size (135 KB). The maximum recommended task size is 100 KB. >> >> The matrix size is 49865 x 191077 and all the singular values are needed. >> >> Is there a way to lift that limit and be able to compute whatever number >> of singular values? >> >> Thank you. >> >> >> >
