[
https://issues.apache.org/jira/browse/MAHOUT-54?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595974#action_12595974
]
Jeremy Chow commented on MAHOUT-54:
-----------------------------------
The reason why I create a new package is that the sequencefile format ouputed
by my implementation has some differences from the original one.
I couldnot syncretize these two swiftly. The key idea of the canopy algorithm
is that one can greatly reduce the number of distance computations required for
clustering by first cheaply partitioning the data into overlapping subsets, and
then only measuring distances among pairs of data points that belong to a
common subset. We assume the distance of two points that never appear in the
same canopy to be infinite. I will add an unit test soon, but you can try my
verson at first following the steps list below .
1. prepare the data points which we will clustering , and the initial k-means
centriods we will take.
This is a fragment of input data points, the format initial centriods according
to is the same with them.
[,125,256,6000,256,16,128,199,]
[,29,8000,32000,32,8,32,253,]
[,29,8000,32000,32,8,32,253,]
[,29,8000,32000,32,8,32,253,]
[,29,8000,16000,32,8,16,132,]
[,26,8000,32000,64,8,32,290,]
[,23,16000,32000,64,16,32,381,]
[,23,16000,32000,64,16,32,381,]
[,23,16000,64000,64,16,32,749,]
[,23,32000,64000,128,32,64,1238,]
[,400,1000,3000,0,1,2,23,]
[,400,512,3500,4,1,6,24,]
[,60,2000,8000,65,1,8,70,]
[,50,4000,16000,65,1,8,117,]
[,350,64,64,0,1,4,15,]
[,200,512,16000,0,4,32,64,]
[,167,524,2000,8,4,15,23,]
[,143,512,5000,0,7,32,29,]
[,143,1000,2000,0,5,16,22,]
[,110,5000,5000,142,8,64,124,]
[,143,1500,6300,0,5,32,35,]
2. create canopies clustering input points, and place the initial k-means
centriods into canopies which are the nearest with them.
bin/hadoop jar apache-mahout-*.jar
org.apache.mahout.clustering.canopykmeans.CanopyKMeansInitialJob input cluster
canopy_output org.apache.mahout.utils.EuclideanDistanceMeasure 5000 3410
this step will produce three folders of datas : canopies , clusters and points.
3. clustering with k-means.
bin/hadoop jar apache-mahout-*.jar
org.apache.mahout.clustering.canopykmeans.CanopyKMeansDriver
canopy_output/points/ canopy_output/clusters/ kmeans_output/
org.apache.mahout.utils.EuclideanDistanceMeasure 0.00001 5
ouput is like these:
V[, 283.5496183206107, 1248.0610687022902, 5003.480916030534,
12.442748091603054, 2.7022900763358777, 13.083969465648854, 36.52671755725191,
] :C0 [, 900.0, 1000.0, 1000.0, 0.0, 1.0, 2.0, 18.0, ]
V[, 283.5496183206107, 1248.0610687022902, 5003.480916030534,
12.442748091603054, 2.7022900763358777, 13.083969465648854, 36.52671755725191,
] :C0 [, 900.0, 1000.0, 4000.0, 4.0, 1.0, 2.0, 26.0, ]
I'm sorry that I am very unacquainted about all conventions in ASF, like the
printing rules, the tabs format, the unsuggestive tags, etc. Would you give me
some stuff about them?
> parallelize k-means sharing the predominance of canopies
> --------------------------------------------------------
>
> Key: MAHOUT-54
> URL: https://issues.apache.org/jira/browse/MAHOUT-54
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.1
> Environment: OS Independent
> Reporter: Jeremy Chow
> Fix For: 0.1
>
> Attachments: canopykeams.patch
>
>
> The implementation of mahout at present only using canopy algorithm creating
> initial cluster centroids for k-means. It will calculate the distance from
> each center to every point while iterating. But the most import improvement
> of canopies is that needs only calculating the distance from each center to
> a much smaller number of points which exists in the same canopy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
