[
https://issues.apache.org/jira/browse/SPARK-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15555385#comment-15555385
]
Seth Hendrickson edited comment on SPARK-17824 at 10/7/16 3:42 PM:
-------------------------------------------------------------------
[~yanboliang] Can you please post your design plans? This is almost certainly
going to conflict with the PR I'm about to send for
[SPARK-17748|https://issues.apache.org/jira/browse/SPARK-17748]. In that PR, I
have implemented a pluggable solver for the normal equations, I posted a bit of
detail on the JIRA. In fact, if it gets merged we will be able to deal with
singular matrices by running L-BFGS on the normal equations on the driver
(one-pass). It may not be the most elegant solution, but it is a byproduct of
implementing the OWL-QN solver. I'd like to hear more about your patch to
understand how the two fit together, what conflicts there are, and how we need
to coordinate.
In fact, I may have already written some of the test cases you will need to
write, so maybe we can share them :)
Thanks!
was (Author: sethah):
[~yanboliang] Can you please post your design plans? This is almost certainly
going to conflict with the PR I'm about to send for
[SPARK-17748|https://issues.apache.org/jira/browse/SPARK-17748]. In that PR, I
have implemented a pluggable solver for the normal equations, I posted a bit of
detail on the JIRA. In fact, if it gets merged we will be able to deal with
singular matrices by running L-BFGS on the normal equations on the driver
(one-pass). It may not be the most elegant solution, but it is a byproduct of
implementing the OWL-QN solver. I'd like to hear more about your patch to
understand how the two fit together, what conflicts there are, and how we need
to coordinate.
Thanks!
> QR solver for WeightedLeastSquares
> ----------------------------------
>
> Key: SPARK-17824
> URL: https://issues.apache.org/jira/browse/SPARK-17824
> Project: Spark
> Issue Type: New Feature
> Components: ML
> Reporter: Yanbo Liang
> Assignee: Yanbo Liang
>
> Cholesky decomposition is unstable (for near-singular and rank deficient
> matrices) and only works on positive definite matrices which can not be
> guaranteed in all cases, it was often used when matrix A is very large and
> sparse due to faster calculation. QR decomposition has better numerical
> properties than Cholesky and can works on matrices which are not positive
> definite. Spark MLlib {{WeightedLeastSquares}} use Cholesky decomposition to
> solve normal equation currently, we should also support or move to QR solver
> for better stability. I'm preparing to send a PR.
> cc [~dbtsai] [~sethah]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
