GitHub user dbtsai opened a pull request:
https://github.com/apache/spark/pull/3565
[SPARK-4708][MLLib] Make k-mean runs two/three times faster with
dense/sparse sample
Note that the usage of `breezeSquaredDistance` in
`org.apache.spark.mllib.util.MLUtils.fastSquaredDistance`
is in the critical path, and breezeSquaredDistance is slow.
We should replace it with our own implementation.
Here is the benchmark against mnist8m dataset.
Before
DenseVector: 70.04secs
SparseVector: 59.05secs
With this PR
DenseVector: 30.58secs
SparseVector: 21.14secs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AlpineNow/spark kmean
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3565.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3565
----
commit 4554ddd7c8a21b3463e3330b41683b943dd3934f
Author: DB Tsai <[email protected]>
Date: 2014-12-03T01:05:06Z
first commit
commit b185a7778f702eff4dc63f801e8ad27f8f318e3f
Author: DB Tsai <[email protected]>
Date: 2014-12-03T01:15:24Z
cleanup
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
