GitHub user etrain opened a pull request:
https://github.com/apache/spark/pull/687
Use numpy directly for matrix multiply.
Using matrix multiply to compute XtX and XtY yields a 5-20x speedup
depending on problem size.
For example - the following takes 19s locally after this change vs. 5m21s
before the change. (16x speedup).
bin/pyspark examples/src/main/python/als.py local[8] 1000 1000 50 10 10
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/etrain/spark-1 patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/687.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #687
----
commit d1ab9b6ef020bc0983a72da8a9add9cfc2356a4c
Author: Evan Sparks <[email protected]>
Date: 2014-05-08T01:18:36Z
Use numpy directly for matrix multiply.
Using matrix multiply to compute XtX and XtY yields a 5-20x speedup
depending on problem size.
For example - the following takes 19s locally after this change vs. 5m21s
before the change. (16x speedup).
bin/pyspark examples/src/main/python/als.py local[8] 1000 1000 50 10 10
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
