Hi, I'm getting java.lang.OutOfMemoryError: Java heap space errors when running nutch in a hadoop cluster. I have doubled the heap by setting export HADOOP_HEAPSIZE=2048 in hadoop-env.sh but this doesn't seem to make a difference.
I'm need to hadoop so appreciate any help. Thanks, Ed. 2008-09-18 14:13:30,274 INFO fetcher.Fetcher - fetch of http://somedomain.com/general/aptrix/aptprop.nsf/Content/CityFlyer+Cabin+Crew+Home%5CLibrary?OpenDocument failed with: java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:30,276 ERROR httpclient.Http - java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:30,635 INFO fetcher.Fetcher - fetch of http://somedomain.com/general/aptrix/aptcsops.nsf/Content/Inflight+Services+Home%5CPeople+%26+Training%5CCrewcare?OpenDocument?OpenDocument failed with: java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:30,635 INFO fetcher.Fetcher - fetch of http://somedomain.com/general/aptrix/apteba.nsf/Content/Commercial+Chat+-+February+7+2008?OpenDocument failed with: java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:30,635 ERROR httpclient.Http - java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:30,636 WARN mapred.TaskTracker - Error running child java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - java.io.IOException: Stream closed 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - java.io.IOException: Stream closed 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1378) 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1337) 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - at java.io.DataInputStream.readInt(DataInputStream.java:353) 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1847) 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1877) 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1378) 2008-09-18 14:13:30,637 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1782) 2008-09-18 14:13:30,638 FATAL fetcher.Fetcher - at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1337) 2008-09-18 14:13:30,638 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1828) 2008-09-18 14:13:30,638 FATAL fetcher.Fetcher - at java.io.DataInputStream.readInt(DataInputStream.java:353) 2008-09-18 14:13:30,638 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79) 2008-09-18 14:13:30,638 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1847) 2008-09-18 14:13:30,638 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:31) 2008-09-18 14:13:30,638 ERROR httpclient.Http - java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:30,638 INFO fetcher.Fetcher - fetch of http://somedomain.com/general/aptrix/aptrix.nsf/Content/Corporate+Principles?OpenDocument failed with: java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:31,048 FATAL fetcher.Fetcher - java.io.IOException: Stream closed 2008-09-18 14:13:31,048 FATAL fetcher.Fetcher - at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1378) 2008-09-18 14:13:31,048 FATAL fetcher.Fetcher - at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1337) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at java.io.DataInputStream.readInt(DataInputStream.java:353) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1847) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1877) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1782) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1828) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:31) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:158) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:127) 2008-09-18 14:13:31,049 FATAL fetcher.Fetcher - fetcher caught:java.io.IOException: Stream closed 2008-09-18 14:13:31,050 FATAL util.LogUtil - Cannot log with method [public abstract void org.apache.commons.logging.Log.fatal(java.lang.Object)] java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.nutch.util.LogUtil$1.flush(LogUtil.java:103) at java.io.PrintStream.write(PrintStream.java:414) at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336) at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:115) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:169) at java.io.PrintStream.newLine(PrintStream.java:478) at java.io.PrintStream.println(PrintStream.java:740) at java.lang.Throwable.printStackTrace(Throwable.java:465) at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:132) Caused by: java.lang.OutOfMemoryError: Java heap space 2008-09-18 14:13:31,050 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1782) 2008-09-18 14:13:31,050 FATAL fetcher.Fetcher - at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1828) 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79) 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:31) 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:158) 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:127) 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - fetcher caught:java.io.IOException: Stream closed 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:158) 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:127) 2008-09-18 14:13:31,051 FATAL fetcher.Fetcher - fetcher caught:java.io.IOException: Stream closed _________________________________________________________________ Make a mini you and download it into Windows Live Messenger http://clk.atdmt.com/UKM/go/111354029/direct/01/
