[ https://issues.apache.org/jira/browse/SPARK-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15555385#comment-15555385 ]
Seth Hendrickson commented on SPARK-17824: ------------------------------------------ [~yanboliang] Can you please post your design plans? This is almost certainly going to conflict with the PR I'm about to send for [SPARK-17748|https://issues.apache.org/jira/browse/SPARK-17748]. In that PR, I have implemented a pluggable solver for the normal equations, I posted a bit of detail on the JIRA. In fact, if it gets merged we will be able to deal with singular matrices by running L-BFGS on the normal equations on the driver (one-pass). It may not be the most elegant solution, but it is a byproduct of implementing the OWL-QN solver. I'd like to hear more about your patch to understand how the two fit together, what conflicts there are, and how we need to coordinate. Thanks! > QR solver for WeightedLeastSquares > ---------------------------------- > > Key: SPARK-17824 > URL: https://issues.apache.org/jira/browse/SPARK-17824 > Project: Spark > Issue Type: New Feature > Components: ML > Reporter: Yanbo Liang > Assignee: Yanbo Liang > > Cholesky decomposition is unstable (for near-singular and rank deficient > matrices) and only works on positive definite matrices which can not be > guaranteed in all cases, it was often used when matrix A is very large and > sparse due to faster calculation. QR decomposition has better numerical > properties than Cholesky and can works on matrices which are not positive > definite. Spark MLlib {{WeightedLeastSquares}} use Cholesky decomposition to > solve normal equation currently, we should also support or move to QR solver > for better stability. I'm preparing to send a PR. > cc [~dbtsai] [~sethah] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org