Hi Urs Did u use Nutch 1.6?
< 2013-05-09 03:57:56,967 WARN fetcher.Fetcher - Attempting to finish item from unknown queue: org.apache.nutch.fetcher. Fetcher$FetchItem@307322f4 > This cause by call the FetchItemQueues#finishFetchItem method, but current queues can not find the queueID of the FetchItem, one reason is that the queue is deleted by reap empty queues in FetchItemQueues#getFetchItem method when that queue is empty, because the FetchItem is unblocked when it crawl finished, but when after that throw an Exception , that FetchItem will be unblock again. but this time, that queue was emptyed. so it will throw this WARN. < 2013-05-09 03:57:56,969 ERROR fetcher.Fetcher - fetcher caught:java.lang. NullPointerException > There are two places can log this exception, one is in Fetcher#output method, another is in Fetcher#run method. see your log order , maybe it is log by Fetcher#run. I can not found any reason that can cause this NPE. Can you reproduce this Exception and provide more detailed log. Thanks On Thu, May 9, 2013 at 6:45 PM, Tejas Patil <[email protected]>wrote: > Hey Urs, > Please see the logs/hadoop.log file and share the stack trace of the > exception. > > > On Thu, May 9, 2013 at 3:36 AM, Urs Hofer <[email protected]> wrote: > > > Dear List > > > > i'm currently running the nutch crawl script with solr4. > > additionally, I'm using the urlmeta plugin and I'm parsing keyword and > > description > > metadata as well. the crawl script runs in local mode. currently, I'm > > seeding about 500 domains. > > > > the crawl script runs without problems the first time. on a recrawl, it > > dies with the error > > > > 2013-05-09 03:57:56,967 WARN fetcher.Fetcher - Attempting to finish item > > from unknown queue: org.apache.nutch.fetcher.Fetcher$FetchItem@307322f4 > > 2013-05-09 03:57:56,967 INFO fetcher.Fetcher - -finishing thread > > FetcherThread, activeThreads=27 > > 2013-05-09 03:57:56,969 INFO fetcher.Fetcher - -finishing thread > > FetcherThread, activeThreads=1 > > 2013-05-09 03:57:56,969 ERROR fetcher.Fetcher - fetcher > > caught:java.lang.NullPointerException > > 2013-05-09 03:57:56,969 INFO fetcher.Fetcher - -finishing thread > > FetcherThread, activeThreads=2 > > 2013-05-09 03:57:56,969 WARN fetcher.Fetcher - Attempting to finish item > > from unknown queue: org.apache.nutch.fetcher.Fetcher$FetchItem@504e3c43 > > 2013-05-09 03:57:56,969 INFO fetcher.Fetcher - -finishing thread > > FetcherThread, activeThreads=0 > > 2013-05-09 03:57:57,942 ERROR fetcher.Fetcher - Fetcher: > > java.io.IOException: Job failed! > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) > > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1332) > > at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:1368) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:1341) > > > > > > after the fetcher part. > > > > I have a very liberal regex-urlfilter configuration: > > > > -^(file|ftp|mailto): > > > > > -\.(mp3|MP3|gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$ > > +. > > > > But I am restricting the crawl to db.ignore.external.links = true > > Can it be because I've removed the line > > > > # skip URLs containing certain characters as probable queries, etc. > > -.*[?*!@=].* > > > > in regex-urlfilter? I got several seed entries like index.php?language=fr > > > > Since I'm new to nutch, I don't know where to search and how to continue. > > Thanks for any help > > Best regards > > Urs Hofer > -- Don't Grow Old, Grow Up... :-)
