Hi, I'm running Mahout LDA on a cluster but I'm getting the java out of memory exception (below). How do I increase the MAHOUT_HEAP variable in bin\mahout.sh ? When I add the line MAHOUT_HEAP=2000 it doesn't seem to change ! Do I have to reload or rebuild anything so that changes take effect ?
thanks. 2011-11-15 16:59:39,697 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2011-11-15 16:59:39,836 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2011-11-15 16:59:39,888 WARN org.apache.hadoop.conf.Configuration: /scratch1/hadoop/mapred/local/taskTracker/mkhooja/jobcache/job_201110181830_3456/job.xml:a attempt to override final parameter: dfs.data.dir; Ignoring. 2011-11-15 16:59:39,956 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100 2011-11-15 16:59:40,001 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720 2011-11-15 16:59:40,001 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2011-11-15 16:59:48,863 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2011-11-15 16:59:48,866 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: Java heap space at org.apache.mahout.math.DenseMatrix.<init>(DenseMatrix.java:51) at org.apache.mahout.clustering.lda.LDAInference.createPhiMatrix(LDAInference.java:154) at org.apache.mahout.clustering.lda.LDAInference.infer(LDAInference.java:93) at org.apache.mahout.clustering.lda.LDAMapper.map(LDAMapper.java:48) at org.apache.mahout.clustering.lda.LDAMapper.map(LDAMapper.java:36) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- M.khouja
