Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15450#discussion_r84455681
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala ---
@@ -378,10 +382,10 @@ class KMeans private (
costs.unpersist(blocking = false)
bcNewCentersList.foreach(_.destroy(false))
- if (centers.size == k) {
+ if (centers.size <= k) {
--- End diff --
That's right, although it's not a goal to guarantee distinct centers,
though that's a nice to have where possible. I think my goal is improving the
straightforward cases, and, maintaining as much consistency as possible. I
agree with adding a test like this and adding a call to `.distinct`. Might as
well take this to a logical conclusion.
It also raises the interesting question: if you have >= k distinct points,
and happen to pick < k distinct centroids, should you go back and replenish the
set of centroids? I am punting on that right now but it's a legitimate point.
It's quite a corner case though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
