[
https://issues.apache.org/jira/browse/SPARK-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-3218:
-----------------------------
Target Version/s: (was: 1.3.0)
> K-Means clusterer can fail on degenerate data
> ---------------------------------------------
>
> Key: SPARK-3218
> URL: https://issues.apache.org/jira/browse/SPARK-3218
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.0.2
> Reporter: Derrick Burns
> Assignee: Derrick Burns
> Labels: clustering
>
> The KMeans parallel implementation selects points to be cluster centers with
> probability weighted by their distance to cluster centers. However, if there
> are fewer than k DISTINCT points in the data set, this approach will fail.
> Further, the recent checkin to work around this problem results in selection
> of the same point repeatedly as a cluster center.
> The fix is to allow fewer than k cluster centers to be selected. This
> requires several changes to the code, as the number of cluster centers is
> woven into the implementation.
> I have a version of the code that addresses this problem, AND generalizes the
> distance metric. However, I see that there are literally hundreds of
> outstanding pull requests. If someone will commit to working with me to
> sponsor the pull request, I will create it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
