srowen commented on a change in pull request #24531: [SPARK-27636][MLLIB]Remove
cached RDD blocks after PIC execution
URL: https://github.com/apache/spark/pull/24531#discussion_r281051280
##########
File path:
mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala
##########
@@ -226,11 +226,10 @@ class PowerIterationClustering private[clustering] (
*/
private def pic(w: Graph[Double, Double]): PowerIterationClusteringModel = {
val v = powerIter(w, maxIterations)
- val assignments = kMeans(v, k).mapPartitions({ iter =>
- iter.map { case (id, cluster) =>
- Assignment(id, cluster)
- }
- }, preservesPartitioning = true)
+ val assignments = kMeans(v, k).map {
+ case (id, cluster) => Assignment(id, cluster)
+ }
Review comment:
Hm yeah that's a good point. I am not sure it's valid to say this RDD
preservers partitioning because the default hash partitioners would has the
tuples and case class differently. The result is not a pair RDD. I don't feel
strongly about it, but i'd keep this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
