The clustering algorithm has also changed internally. So, expect the
results to be different ( and better ).
I can think of one reason for this behavior. Maybe lots of clusters are
having only one vector inside it, and, AFAIK, clusterdumper will not
output any cluster with single vector.
So, I think, its clusterdumper which is doing the invisible "pruning" (
by not ouputting clusters with single vectors ).
Can you cross check the output once with ClusterOutputPostProcessorDriver?
No, no tool can output the pruned vectors. The only way to see all
vectors assigned to any cluster is to set clusterClassificationThreshold
to 0.
If you still face the problem, then please provide the parameters with
which you are calling kmeans.
Regarding "I should also mention I have vectors which are exactly the
same (even their names), perhaps they are the ones being pruned, is that
possible? "
The name of the vector has nothing to do with clustering, I am not sure
whether it will have any effect when clusterdumper is in action. So,
crosschecking with ClusterOutputPostProcessorDriver will answer this.
Good luck.
Paritosh
On 17-08-2012 21:07, Whitmore, Mattie wrote:
Sure, I have a dataset which I wish to cluster using Kmeans. Previously (v0.5)
when I did a clusterdump the total amount of vectors within the resultant
clusters was the same as the total amount fed to the algorithm. I wish this to
be the case when clustering with v0.7. The only change in the algorithm is
clusterClassificationThreshold, I set this value to be 0 so that it will in
fact cluster all vectors in the dataset.
My logic here was no vector should have a probability of being in some cluster
less than 0 and therefore all vectors should cluster.
However after running a clusterdump I find that vectors (1/3 roughly) have been
pruned.
Is this a bug, or me just not understanding the new capabilities?
I should also mention I have vectors which are exactly the same (even their
names), perhaps they are the ones being pruned, is that possible?
Another question if I may: I will eventually want to use the pruning
capabilities, does the ClusterOutputPostProcessorDriver method (or a similar
method) have the capability of outputting the pruned vectors into a folder?
Thanks! Please let me know if I'm still not being clear enough.
Mattie
-----Original Message-----
From: Paritosh Ranjan [mailto:[email protected]]
Sent: Friday, August 17, 2012 11:20 AM
To: [email protected]
Subject: Re: Mahout-279/kmeans++
clusterClassificationThreshold is for outlier removal, and this is the way it
should be used.
Can you provide some more information about your job and the way you are
calling it?
And if I look at the code, the vector should be clustered even if the pdf is 0.
The method which decides whether the vector should be assigned to a particular
cluster or not -
/**
* Decides whether the vector should be classified or not based on the max
pdf
* value of the clusters and threshold value.
*
* @return whether the vector should be classified or not.
*/
private static boolean shouldClassify(Vector pdfPerCluster, Double
clusterClassificationThreshold) {
return pdfPerCluster.maxValue() >= clusterClassificationThreshold;
}
On 17-08-2012 20:06, Whitmore, Mattie wrote:
Hi Ted,
Yes this is great! I hope to start working with this algorithm in the next
couple weeks.
I have a question about the 0.7 implementation of kmeans and the
clusterClassificationThreshold, I have this value set at zero, but the output
is still showing that about 1/3 of my data is not assigned to a cluster in my
output. Am I using this value incorrectly? I did a kmeansdriver.run with the
0.5 and 0.7 api, and had the data pruned despite the
clusterClassificationThreshold = 0.
Thanks,
Mattie
-----Original Message-----
From: Ted Dunning [mailto:[email protected]]
Sent: Wednesday, August 15, 2012 5:20 PM
To: [email protected]
Subject: Re: Mahout-279/kmeans++
Mattie,
Would this help?
https://github.com/tdunning/knn/blob/master/src/main/java/org/apache/mahout/knn/cluster/BallKmeans.java
and
https://github.com/tdunning/knn/blob/master/docs/scaling-k-means/scaling-k-means.pdf
On Wed, Aug 15, 2012 at 10:45 AM, Whitmore, Mattie <[email protected]>wrote:
Hi!
I have been using RandomSeedGenerator, and was hoping it had a patch like
that described in Mahout-279 since I want only 10 vectors out of a set of
more than 100,000,000. I have been using canopy clustering for better
results, but still need to do a few passes of kmeans to determine my T, and
the random seed does take a long time.
The comments say that you are working on a kmeans++, I searched around but
couldn't confirm any more information about it. Is a scalable kmeans++ in
the works? (I know research on the subject is quite new)
Thanks!
Mattie Whitmore
Mathematician/IR&D Software Engineer
HARRIS Corporation - Advanced Information Solutions
301.837.5278
[email protected]<mailto:[email protected]>
