[ https://issues.apache.org/jira/browse/SPARK-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
yuhao yang closed SPARK-5406. ----------------------------- fix and merged. Thanks > LocalLAPACK mode in RowMatrix.computeSVD should have much smaller upper bound > ----------------------------------------------------------------------------- > > Key: SPARK-5406 > URL: https://issues.apache.org/jira/browse/SPARK-5406 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.2.0 > Environment: centos, others should be similar > Reporter: yuhao yang > Assignee: yuhao yang > Priority: Minor > Fix For: 1.3.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > In RowMatrix.computeSVD, under LocalLAPACK mode, the code would invoke > brzSvd. Yet breeze svd for dense matrix has latent constraint. In it's > implementation > ( > https://github.com/scalanlp/breeze/blob/master/math/src/main/scala/breeze/linalg/functions/svd.scala > ): > val workSize = ( 3 > * scala.math.min(m, n) > * scala.math.min(m, n) > + scala.math.max(scala.math.max(m, n), 4 * scala.math.min(m, n) > * scala.math.min(m, n) + 4 * scala.math.min(m, n)) > ) > val work = new Array[Double](workSize) > as a result, column num must satisfy 7 * n * n + 4 * n < Int.MaxValue > thus, n < 17515. > This jira is only the first step. If possbile, I hope spark can handle matrix > computation up to 80K * 80K. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org