Github user debasish83 commented on the pull request:
https://github.com/apache/spark/pull/5005#issuecomment-83752607
@tmyklebu these least squares problem need not be necessarily small but for
mllib ALS they are...
Think about TRON (breeze.optimize.TruncatedNewtonMinimizer) and the
underlying CG solver in TRON which is very similar to NNLS...There also we use
a Projected Conjugate Gradient Solver and solve large problems....Also the
direct/interior point based solvers are more robust to condition number as long
as you can represent the gram matrix as sparse matrix and use sparse algebra..
I think for mllib ALS, it's just a design decision that whether we want to
give all the intermediate state to the user or just the last state...To
optimize the runtime for direct solvers in breeze we can do the following:
1. Take a initialState from user
2. Return a finalState to user
3. Run a while loop inside in place of iterator
If you guys agree I can do this change to both NNLS and QuadraticMinimizer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
