GitHub user hhbyyh opened a pull request:
https://github.com/apache/spark/pull/5909
[SPARK-7368][MLlib] Add QR decomposition for RowMatrix
jira: https://issues.apache.org/jira/browse/SPARK-7368
Add QR decomposition for RowMatrix.
I'm not sure what's the blueprint about the distributed Matrix from
community and whether this will be a desirable feature , so I sent a prototype
for discussion. I'll go on polish the code and provide ut and performance
statistics if it's acceptable.
The implementation refers to the [paper:
https://www.cs.purdue.edu/homes/dgleich/publications/Constantine%202011%20-%20TSQR.pdf]
Austin R. Benson, David F. Gleich, James Demmel. "Direct QR factorizations
for tall-and-skinny matrices in MapReduce architectures", 2013 IEEE
International Conference on Big Data,
which is a stable algorithm with good scalability.
Currently I tried it on a 400000 * 500 rowMatrix and it can bring down the
computation time from 8.8 mins (using breeze.linalg.qr.reduced) to 2.6 mins on
a 4 worker cluster. I think there will still be some root for performance
improvement.
Any trial and suggestion is welcome.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hhbyyh/spark qrDecomposition
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5909.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5909
----
commit 39b0b227d3bedfc789743972f5a0ded8583e8e7b
Author: Yuhao Yang <[email protected]>
Date: 2015-05-05T11:51:51Z
initial draft for discussion
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
