Caching after doing the multiply is a good idea. Keep in mind that during
the first iteration of KMeans, the cached rows haven't yet been
materialized - so it is both doing the multiply and the first pass of
KMeans all at once. To isolate which part is slow you can run
cachedRows.numRows() to force this to be materialized before you run KMeans.
Also, KMeans is optimized to run quickly on both sparse and dense data. The
result of PCA is going to be dense, but if your input data has #nnz ~=
size(pca data), performance might be about the same. (I haven't actually
verified this last point.)
Finally, speed is partially going to be dependent on how much data you have
relative to scheduler overheads - if your input data is small it could be
that the costs of distributing your task are greater than the time spent
actually computing - usually this would manifest itself in the stages
taking about the same amount of time even though you're passing datasets
that have different dimensionality.
On Tue, Sep 30, 2014 at 9:00 AM, st553 sthompson...@gmail.com wrote:
Thanks for your response Burak it was very helpful.
I am noticing that if I run PCA before KMeans that the KMeans algorithm
will
actually take longer to run than if I had just run KMeans without PCA. I
was
hoping that by using PCA first it would actually speed up the KMeans
algorithm.
I have followed the steps you've outlined but Im wondering if I need to
cache/persist the RDD[Vector] rows of the RowMatrix returned after
multiplying. Something like:
val newData: RowMatrix = data.multiply(bcPrincipalComponents.value)
val cachedRows = newData.rows.persist()
KMeans.run(cachedRows)
cachedRows.unpersist()
It doesnt seem intuitive to me that a smaller dimensional version of my
data
set would take longer for KMeans... unless Im missing something?
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-run-kmeans-after-pca-tp14473p15409.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
