Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11610#issuecomment-197557069
@dbtsai There is a good chance of precision loss during the computation of
A^T A is A is ill-conditioned. A better approach is to factorize A directly. It
is similar to tall-skinny QR without storing Q (applying Q^T to be directly).
SVD is similar. See this paper:
http://web.stanford.edu/~paulcon/docs/mapreduce-2013-arbenson.pdf. We can
definitely switch to it to get better stability but we need to handle sparsity,
which might not be worth the time.
@iyounus You can use `RCOND` to control the rank estimation. Usually a
number like `1e-12` should work well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
