hi,
i am running on a single machine; 2G RAM, and java heap space set at
1024m, the segments are quite - tiny less than 100 urls and during
mergeSegments i get this exception below;
i have set mapred.child.java.opts=-Xmx512m but there is no change;
any suggestions?
====>
2009-11-03 17:58:28,971 INFO [org.apache.hadoop.mapred.LocalJobRunner]
reduce > reduce
2009-11-03 17:58:38,448 INFO [org.apache.hadoop.mapred.LocalJobRunner]
reduce > reduce
2009-11-03 17:58:57,085 INFO [org.apache.hadoop.mapred.LocalJobRunner]
reduce > reduce
2009-11-03 17:59:34,723 INFO [org.apache.hadoop.mapred.LocalJobRunner]
reduce > reduce
2009-11-03 18:02:09,660 INFO [org.apache.hadoop.mapred.TaskRunner]
Communication exception: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.Counters
$Group.getCounterForName(Counters.java:327)
at
org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:494)
at org.apache.hadoop.mapred.Counters.sum(Counters.java:506)
at org.apache.hadoop.mapred.LocalJobRunner
$Job.statusUpdate(LocalJobRunner.java:222)
at org.apache.hadoop.mapred.Task$1.run(Task.java:418)
at java.lang.Thread.run(Thread.java:619)
2009-11-03 18:02:10,376 WARN [org.apache.hadoop.mapred.LocalJobRunner]
job_local_0001
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:715)
at
org.apache.hadoop.mapred.LocalJobRunner.killJob(LocalJobRunner.java:310)
at org.apache.hadoop.mapred.JobClient
$NetworkedJob.killJob(JobClient.java:315)
at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1239)
at
org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:620)
at
org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:665)
