Hans Brende created NUTCH-2551: ---------------------------------- Summary: NullPointerException in generator Key: NUTCH-2551 URL: https://issues.apache.org/jira/browse/NUTCH-2551 Project: Nutch Issue Type: Bug Components: generator Affects Versions: 1.15 Reporter: Hans Brende Fix For: 1.15
A NullPointerException is thrown during the crawl generate stage when I deploy to a hadoop cluster (but for some reason, it works fine locally). It looks like this is caused because the URLPartitioner class still has the old {{configure()}} method in there (which is never called, causing the {{normalizers}} field to remain null), rather than implementing the {{Configurable}} interface as detailed in the newer mapreduce API's Partitioner spec. Stack trace: {code} java.lang.NullPointerException at org.apache.nutch.crawl.URLPartitioner.getPartition(URLPartitioner.java:76) at org.apache.nutch.crawl.URLPartitioner.getPartition(URLPartitioner.java:40) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:716) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.nutch.crawl.Generator$SelectorInverseMapper.map(Generator.java:553) at org.apache.nutch.crawl.Generator$SelectorInverseMapper.map(Generator.java:546) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)