if you set the mapred.child.java.opts with additional value "-XX: -UseGCOverheadLimit" you can bypass this exception. I don't know if it has any side effects as a result of this.. ex. -Xmx512m -XX: -UseGCOverheadLimit
On Tue, Nov 3, 2009 at 7:50 AM, Fadzi Ushewokunze <fa...@butterflycluster.net> wrote: > hi, > > i am running on a single machine; 2G RAM, and java heap space set at > 1024m, the segments are quite - tiny less than 100 urls and during > mergeSegments i get this exception below; > > i have set mapred.child.java.opts=-Xmx512m but there is no change; > > any suggestions? > > > ====> > > 2009-11-03 17:58:28,971 INFO [org.apache.hadoop.mapred.LocalJobRunner] > reduce > reduce > 2009-11-03 17:58:38,448 INFO [org.apache.hadoop.mapred.LocalJobRunner] > reduce > reduce > 2009-11-03 17:58:57,085 INFO [org.apache.hadoop.mapred.LocalJobRunner] > reduce > reduce > 2009-11-03 17:59:34,723 INFO [org.apache.hadoop.mapred.LocalJobRunner] > reduce > reduce > 2009-11-03 18:02:09,660 INFO [org.apache.hadoop.mapred.TaskRunner] > Communication exception: java.lang.OutOfMemoryError: Java heap space > at org.apache.hadoop.mapred.Counters > $Group.getCounterForName(Counters.java:327) > at > org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:494) > at org.apache.hadoop.mapred.Counters.sum(Counters.java:506) > at org.apache.hadoop.mapred.LocalJobRunner > $Job.statusUpdate(LocalJobRunner.java:222) > at org.apache.hadoop.mapred.Task$1.run(Task.java:418) > at java.lang.Thread.run(Thread.java:619) > > 2009-11-03 18:02:10,376 WARN [org.apache.hadoop.mapred.LocalJobRunner] > job_local_0001 > java.lang.ThreadDeath > at java.lang.Thread.stop(Thread.java:715) > at > org.apache.hadoop.mapred.LocalJobRunner.killJob(LocalJobRunner.java:310) > at org.apache.hadoop.mapred.JobClient > $NetworkedJob.killJob(JobClient.java:315) > at > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1239) > at > org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:620) > at > org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:665) > >
