For visualization, you can also do sampling to keep no more than, say, 1000 points per cluster. If you also remember what fraction of the points you have kept, you can plot the points with different transparency to get a good visual rendition of the clusters.
On Mon, Oct 31, 2011 at 9:17 AM, Jeff Eastman <[email protected]> wrote: > Unfortunately, the cluster dumper loads all the points into memory so it > can sort them by cluster for display. What are you trying to do with the > 20M points? Certainly not display them! A better step for subsequent > processing would be for you to write a short MR program to read in the > clusteredPoints directory and output each point to its clustered (mapper). > Then, using k reducers, each of which will get the points for one cluster, > each reducer will output a folder containing all the points for that > cluster. > > -----Original Message----- > From: Vitaly Davydov [mailto:[email protected]] > Sent: Monday, October 31, 2011 4:21 AM > To: [email protected] > Subject: ClusterDumper issue > > Hello, > We use mahout Kmeans-algorithm and convert its binary output to text > representation via ClusterDumper. When our input has reached approximately > 20 million points, ClusterDumper takes 2.5 Gb RAM and fails with "Out of > memory error". Our machines don't have swap and we can't increase RAM > currently. Is there a way to avoid this problem? > > The exception is attached below: > > "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > > org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:101) > at > > org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:38) > at > > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) > at > > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) > at com.google.common.collect.Iterators$5.hasNext(Iterators.java:474) > at > > com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:39) > at > > org.apache.mahout.utils.clustering.ClusterDumper.readPoints(ClusterDumper.java:239) > at > > org.apache.mahout.utils.clustering.ClusterDumper.init(ClusterDumper.java:193) > at > > org.apache.mahout.utils.clustering.ClusterDumper.<init>(ClusterDumper.java:78) > at com.mirantis.bigdata.clustering.kmeans.KmeansJob.run(Unknown Source) > at com.mirantis.bigdata.clustering.kmeans.KmeansJob.main(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:186)" > > -- > Regards, > Vitaly Davydov >
