GitHub user ZakariaHili reopened a pull request:
https://github.com/apache/spark/pull/15965
[SPARK-18356] [ML] Improve MLKmeans Performance
## What changes were proposed in this pull request?
Spark Kmeans fit() doesn't cache the RDD which generates a lot of warnings
:
WARN KMeans: The input data is not directly cached, which may hurt
performance if its parent RDDs are also uncached.
So, Kmeans should cache the internal rdd before calling the Mllib.Kmeans
algo, this helped to improve spark kmeans performance by 14%
https://github.com/ZakariaHili/spark/commit/a9cf905cf7dbd50eeb9a8b4f891f2f41ea672472
@hhbyyh
## How was this patch tested?
Pass Kmeans tests and existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ZakariaHili/spark zakbranch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15965.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15965
----
commit 6b596dcd6a8fe6a6574e4fdda80fdf424c7d8777
Author: Zakaria_Hili <[email protected]>
Date: 2016-11-22T09:33:11Z
[SPARK-18356] [ML] Improve MLKmeans Performance
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
