I've just upgraded to Nutch 0.9.0 and am getting a NullPointerException during my crawl. I turned on lots of logging for nutch and hadoop. Nutch is able to complete a smaller crawl of our intranet without a problem. This only crops up when doing the full crawl. Here is the relevant section from the logs (sorry for any excess junk, I wasn't sure how much would be helpful): 2007-05-10 18:47:25,862 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, 2007-05-10 18:47:26,907 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, 2007-05-10 18:47:28,290 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, fetching http://machine/file.txt 2007-05-10 18:47:28,613 INFO fetcher.Fetcher (Fetcher.java:run(134)) - fetching http://machine/file.txt 2007-05-10 18:47:29,299 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, 2007-05-10 18:47:30,566 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, 2007-05-10 18:47:32,530 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, 2007-05-10 18:47:34,021 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, 2007-05-10 18:47:35,287 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, 2007-05-10 18:47:36,447 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - 117592 pages, 99 errors, 2.4 pages/s, 649 kb/s, java.lang.NullPointerException Exception in thread "main" java.io.IOException: Job failed! 2007-05-10 18:47:39,700 FATAL fetcher.Fetcher (?:invoke0(?)) - java.lang.NullPointerException at org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(FSDataInputStream. java:87) 2007-05-10 18:47:40,203 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(FSDataInputStream. java:87) at org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:12 5) 2007-05-10 18:47:40,205 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:12 5) at org.apache.hadoop.io.SequenceFile$Reader.getPosition(SequenceFile.java: 1736) 2007-05-10 18:47:40,206 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.io.SequenceFile$Reader.getPosition(SequenceFile.java: 1736) at org.apache.hadoop.mapred.SequenceFileRecordReader.getProgress(SequenceF ileRecordReader.java:108) 2007-05-10 18:47:40,206 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.mapred.SequenceFileRecordReader.getProgress(SequenceF ileRecordReader.java:108) at org.apache.hadoop.mapred.MapTask$1.getProgress(MapTask.java:165) 2007-05-10 18:47:40,207 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.mapred.MapTask$1.getProgress(MapTask.java:165) at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:155) 2007-05-10 18:47:40,208 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:155) at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:115) 2007-05-10 18:47:40,208 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:115) fetcher caught:java.lang.NullPointerException 2007-05-10 18:47:40,255 FATAL fetcher.Fetcher (Fetcher.java:run(264)) - fetcher caught:java.lang.NullPointerException at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604) at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:470) at org.apache.nutch.crawl.Crawl.main(Crawl.java:124) java.lang.NullPointerException 2007-05-10 18:47:41,533 FATAL fetcher.Fetcher (?:invoke0(?)) - java.lang.NullPointerException at org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(FSDataInputStream. java:87) 2007-05-10 18:47:41,534 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(FSDataInputStream. java:87) at org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:12 5) 2007-05-10 18:47:41,534 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:12 5) at org.apache.hadoop.io.SequenceFile$Reader.getPosition(SequenceFile.java: 1736) 2007-05-10 18:47:41,535 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.io.SequenceFile$Reader.getPosition(SequenceFile.java: 1736) at org.apache.hadoop.mapred.SequenceFileRecordReader.getProgress(SequenceF ileRecordReader.java:108) 2007-05-10 18:47:41,535 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.mapred.SequenceFileRecordReader.getProgress(SequenceF ileRecordReader.java:108) at org.apache.hadoop.mapred.MapTask$1.getProgress(MapTask.java:165) 2007-05-10 18:47:41,535 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.mapred.MapTask$1.getProgress(MapTask.java:165) at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:155) 2007-05-10 18:47:41,536 FATAL fetcher.Fetcher (?:invoke0(?)) - at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:155) java.lang.NullPointerException at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:115) java.lang.NullPointerException If anyone has any ideas on what I'm doing wrong, please let me know. Thanks so much. Jeff
