I am clustering binary data (feature vaues are 0 or 1) over 20k points
with 200k columns. I use canopy to find initial clusters and then do
kmeans using Manhattan distance in 10 iterations.
After clustering I found that there are many clusters with just one
point and a few very large clusters. I draw the similarity matrix of
clusters (not centroid but OR of bits for points in each cluster but as
most of clusters have only 1 point this is the same as centroid). It
shows that there is a kind of pattern in similarity of matrices.
http://enl.usc.edu/~moshref/cluster_100.jpg
I also run clustering with fewer clusters (by increasing the canopy t2
threshold) and the same pattern occurs.
http://enl.usc.edu/~moshref/cluster_200.jpg
- Am I doing something wrong?
- I want to find uniform size clusters, is kmeans enough for that? is
hierarchical method good for this goal? why?
The definition of size of cluster is the number of 1 bits when we OR all
bits for members of a cluster. However, the number of points in each
cluster may also work
Thanks in advance
--
Masoud Moshref Javadi
Computer Engineering PhD Student
Ming Hsieh Department of Electrical Engineering
University of Southern California
