[
https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356399#comment-14356399
]
Yu Ishikawa commented on SPARK-2429:
------------------------------------
[~rnowling]
I apologize for the delay in replying. I'm still working on this.
I will reply the feedback from Freeman and Owen ASSP.
By the way, I have a question about the future course of action.
I implemented another algorithm which is more scalable and 1000 times faster
than current one.
Should we continue the PR, or replace it with the new implementation?
https://github.com/yu-iskw/more-scalable-hierarchical-clustering-with-spark
It is difficult to run the current one with an argument of the number of large
clusters, such as 10000. Because the dividing processes are executed
one-by-one. The dividing processes of the new one is parallel processing.
That's because it is more scalable and much faster than the current one.
thanks
> Hierarchical Implementation of KMeans
> -------------------------------------
>
> Key: SPARK-2429
> URL: https://issues.apache.org/jira/browse/SPARK-2429
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: RJ Nowling
> Assignee: Yu Ishikawa
> Priority: Minor
> Labels: clustering
> Attachments: 2014-10-20_divisive-hierarchical-clustering.pdf, The
> Result of Benchmarking a Hierarchical Clustering.pdf,
> benchmark-result.2014-10-29.html, benchmark2.html
>
>
> Hierarchical clustering algorithms are widely used and would make a nice
> addition to MLlib. Clustering algorithms are useful for determining
> relationships between clusters as well as offering faster assignment.
> Discussion on the dev list suggested the following possible approaches:
> * Top down, recursive application of KMeans
> * Reuse DecisionTree implementation with different objective function
> * Hierarchical SVD
> It was also suggested that support for distance metrics other than Euclidean
> such as negative dot or cosine are necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
