[ https://issues.apache.org/jira/browse/NUTCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659958#action_12659958 ]
Todd Lipcon commented on NUTCH-669: ----------------------------------- Found the exception in a screen log: {noformat} java.lang.NullPointerException at org.apache.nutch.crawl.MapWritable$KeyValueEntry.access$102(MapWritable.java:469) at org.apache.nutch.crawl.MapWritable.readFields(MapWritable.java:362) at org.apache.nutch.crawl.CrawlDatum.readFields(CrawlDatum.java:250) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817) at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790) at org.apache.hadoop.mapred.SequenceFileRecordReader.getCurrentValue(SequenceFileRecordReader.java:103) at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:78) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170) at org.apache.nutch.fetcher.Fetcher$FetchMapper.run(Fetcher.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child.main(Child.java:155) {noformat} I think NUTCH-676 may help this. Trying another run in a minute. > Consolidate code for Fetcher and Fetcher2 > ----------------------------------------- > > Key: NUTCH-669 > URL: https://issues.apache.org/jira/browse/NUTCH-669 > Project: Nutch > Issue Type: Improvement > Components: fetcher > Affects Versions: 0.9.0 > Reporter: Todd Lipcon > Fix For: 1.0.0 > > > I'd like to consolidate a lot of the common code between Fetcher and > Fetcher2.java. > It seems to me like there are the following differences: > - Fetcher relies on the Protocol to obey robots.txt and crawl delay > settings whereas Fetcher2 implements them itself > - Fetcher2 uses a different queueing model (queue per crawl host) to > accomplish the per-host limiting without making the Protocol do it. > I've begun work on this but want to check with people on the following: > - What reason is there for Fetcher existing at all since Fetcher2 seems to be > a superset of functionality? > - Is it on the road map to remove the robots/delay logic from the Http > protocol and make Fetcher2's delegation of duties the standard? > - Any other improvements wanted for Fetcher while I am in and around the code? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.