Yash Thenuan created NUTCH-2526:
-----------------------------------

             Summary: scoring-opic creating Issues while indexing some 
documents which were generated at parsetime.
                 Key: NUTCH-2526
                 URL: https://issues.apache.org/jira/browse/NUTCH-2526
             Project: Nutch
          Issue Type: Improvement
          Components: parser, scoring
    Affects Versions: 1.14
            Reporter: Yash Thenuan
             Fix For: 1.15


I was trying to write a parse filter plugin whose work was to parse internal 
links as a separate document.what I did basically is,breaking the page into 
multiple parseResults each parseResult having ParseText and ParseData 
corresponding to the InternalLinks. I was successfully able to parse them 
separately. But at the time of Scoring Some Error occurred.

I am attaching the Logs for Indexing.

 

 2018-03-07 15:41:52,327 INFO  indexer.IndexerMapReduce - IndexerMapReduce: 
crawldb: crawl/crawldb
2018-03-07 15:41:52,327 INFO  indexer.IndexerMapReduce - IndexerMapReduce: 
linkdb: crawl/linkdb
2018-03-07 15:41:52,327 INFO  indexer.IndexerMapReduce - IndexerMapReduces: 
adding segment: crawl/segments/20180307130959
2018-03-07 15:41:53,677 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2018-03-07 15:41:54,861 INFO  indexer.IndexWriters - Adding 
org.apache.nutch.indexwriter.elasticrest.ElasticRestIndexWriter
2018-03-07 15:41:55,168 INFO  client.AbstractJestClient - Setting server pool 
to a list of 1 servers: [http://localhost:9200]
2018-03-07 15:41:55,170 INFO  client.JestClientFactory - Using multi 
thread/connection supporting pooling connection manager
2018-03-07 15:41:55,238 INFO  client.JestClientFactory - Using default GSON 
instance
2018-03-07 15:41:55,238 INFO  client.JestClientFactory - Node Discovery 
disabled...
2018-03-07 15:41:55,238 INFO  client.JestClientFactory - Idle connection 
reaping disabled...
2018-03-07 15:41:55,282 INFO  elasticrest.ElasticRestIndexWriter - Processing 
remaining requests [docs = 1, length = 210402, total docs = 1]
2018-03-07 15:41:55,361 INFO  elasticrest.ElasticRestIndexWriter - Processing 
to finalize last execute
2018-03-07 15:41:55,458 INFO  elasticrest.ElasticRestIndexWriter - Previous 
took in ms 175, including wait 97
2018-03-07 15:41:55,468 WARN  mapred.LocalJobRunner - job_local1561152089_0001
java.lang.Exception: java.lang.NullPointerException
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.NullPointerException
        at 
org.apache.nutch.scoring.opic.OPICScoringFilter.indexerScore(OPICScoringFilter.java:171)
        at 
org.apache.nutch.scoring.ScoringFilters.indexerScore(ScoringFilters.java:120)
        at 
org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:296)
        at 
org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:57)
        at 
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-03-07 15:41:55,510 ERROR indexer.IndexingJob - Indexer: 
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873)
        at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:147)
        at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:230)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:239)




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to