GitHub user ZakariaHili opened a pull request:
https://github.com/apache/spark/pull/15965
[SPARK-18356] [ML] Improve MLKmeans Performance
## What changes were proposed in this pull request?
Spark Kmeans fit() doesn't cache the RDD which generates a lot of warnings
:
WARN KMeans: The input data is not directly cached, which may hurt
performance if its parent RDDs are also uncached.
So, Kmeans should cache the internal rdd before calling the Mllib.Kmeans
algo, this helped to improve spark kmeans performance by 14%
https://github.com/ZakariaHili/spark/commit/a9cf905cf7dbd50eeb9a8b4f891f2f41ea672472
@hhbyyh
## How was this patch tested?
Pass Kmeans tests and existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ZakariaHili/spark zakbranch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15965.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15965
----
commit a9cf905cf7dbd50eeb9a8b4f891f2f41ea672472
Author: Zakaria_Hili <[email protected]>
Date: 2016-11-21T13:36:47Z
improve MLKmeans
commit f83473470b09cc52b3ebdc3f71c381659ff096d0
Author: HILI Zakaria <[email protected]>
Date: 2016-11-21T13:48:22Z
[SPARK-18356] [ML] Improve MLKmeans Performance
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
