i use the mahout 4.0 release.
in mahout-distribution-0.4/bin, i used
./mahout canopy -i /home/space/lucene_clustering/vector/vector -o
/home/space/lucene_clustering/canopy/ -dm
org.apache.mahout.common.distance.EuclideanDistanceMeasure -t1 0.8 -t2 0.2 -ow
int hadoop-env.sh, i add the
export HADOOP_HEAPSIZE=20000
export HADOOP_OPTS="-Xmn3g -Xss128k -XX:ParallelGCThreads=20
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:+AggressiveOpts
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false"
i am sure all the parameters are workable, because i use the jconsole to check
the vm paramters.
however, after map 100% and reduce 100%, the memory increase from 2.5G to 20G
and the exception thrown. the file vector is 30m, 50000 records, which is used
for canopy.
10/11/23 16:04:27 INFO mapred.LocalJobRunner: reduce > reduce
10/11/23 16:04:27 INFO mapred.JobClient: map 100% reduce 100%
10/11/23 16:04:30 INFO mapred.LocalJobRunner: reduce > reduce
10/11/23 16:08:17 WARN mapred.LocalJobRunner: job_local_0001
java.lang.OutOfMemoryError: Java heap space
at
org.apache.mahout.math.map.OpenIntDoubleHashMap.rehash(OpenIntDoubleHashMap.java:434)
at
org.apache.mahout.math.map.OpenIntDoubleHashMap.put(OpenIntDoubleHashMap.java:387)
at
org.apache.mahout.math.RandomAccessSparseVector.setQuick(RandomAccessSparseVector.java:134)
at org.apache.mahout.math.AbstractVector.assign(AbstractVector.java:449)
at
org.apache.mahout.clustering.AbstractCluster.computeParameters(AbstractCluster.java:184)
at
org.apache.mahout.clustering.canopy.CanopyReducer.reduce(CanopyReducer.java:42)
at
org.apache.mahout.clustering.canopy.CanopyReducer.reduce(CanopyReducer.java:29)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
10/11/23 16:08:18 INFO mapred.JobClient: Job complete: job_local_0001
10/11/23 16:08:18 INFO mapred.JobClient: Counters: 12
10/11/23 16:08:18 INFO mapred.JobClient: FileSystemCounters
10/11/23 16:08:18 INFO mapred.JobClient: FILE_BYTES_READ=70413991
10/11/23 16:08:18 INFO mapred.JobClient: FILE_BYTES_WRITTEN=164338288
10/11/23 16:08:18 INFO mapred.JobClient: Map-Reduce Framework
10/11/23 16:08:18 INFO mapred.JobClient: Reduce input groups=1
10/11/23 16:08:18 INFO mapred.JobClient: Combine output records=0
10/11/23 16:08:18 INFO mapred.JobClient: Map input records=50000
10/11/23 16:08:18 INFO mapred.JobClient: Reduce shuffle bytes=0
10/11/23 16:08:18 INFO mapred.JobClient: Reduce output records=227
10/11/23 16:08:18 INFO mapred.JobClient: Spilled Records=64708
10/11/23 16:08:18 INFO mapred.JobClient: Map output bytes=8836211
10/11/23 16:08:18 INFO mapred.JobClient: Combine input records=0
10/11/23 16:08:18 INFO mapred.JobClient: Map output records=32354
10/11/23 16:08:18 INFO mapred.JobClient: Reduce input records=32354
Exception in thread "main" java.lang.InterruptedException: Canopy Job failed
processing /home/space/lucene_clustering/vector/vector
at
org.apache.mahout.clustering.canopy.CanopyDriver.buildClustersMR(CanopyDriver.java:252)
at
org.apache.mahout.clustering.canopy.CanopyDriver.buildClusters(CanopyDriver.java:167)
at
org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:114)
at
org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:91)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.mahout.clustering.canopy.CanopyDriver.main(CanopyDriver.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
