The property fetcher.parse is false and I pass -noParsing to the fetch command.
What other post-fetch actions are there?
From: Sebastian Nagel <[email protected]>
To: [email protected]
Sent: Friday, December 9, 2016 12:58 AM
Subject: Re: Fetcher "hung while processing"
Hi Michael,
what about the property fetcher.parse ?
The queue is unblocked after a page has been fetched but before parsing.
If the parser is hanging or one of the post-fetch actions take too long
it may happen that there are multiple URLs from the same host still in
process.
Sebastian
On 12/09/2016 02:15 AM, Michael Coffey wrote:
> I sometimes get a bunch of warning messages that say Thread #x hung while
> processing <url>
> Is this just a normal thing to see occasionally, or should I look to find
> some resolution? I do have an example where the same host shows up on a
> multitude of these messages, which puzzles me. I think there should be only
> one thread per host, due to me specifying fetcher.threads.per.queue=1
> Here is example log showing the first 20 of 50 hung threads. Note that
> http://shinystat.com and http://fabulous.com show up more than once.
>
> 2016-12-09 00:47:29,559 WARN [main] org.apache.nutch.fetcher.Fetcher:
> Aborting with 50 hung threads.
> 2016-12-09 00:47:29,560 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #0 hung while processing
> https://www.hugedomains.com/domain_search.cfm?catSearch=434
> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #1 hung while processing
> http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=820
> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #2 hung while processing http://shinystat.com/it/pro/info_pro.html
> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #3 hung while processing http://events.stanford.edu/byCategory/13/
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #4 hung while processing
> https://www.ladesk.com/pricing/hosted/terms-and-conditions/
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #5 hung while processing http://shinystat.com/en/opt-out_free.html
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #6 hung while processing http://shinystat.com/fr/biz/info_biz.html
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #7 hung while processing
> http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=88
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #8 hung while processing http://www.youronlinechoices.com/sk/slovnik-pojmov
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #9 hung while processing https://twitter.com/sakura_ope
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #10 hung while processing http://europa.eu/european-union/topics/culture_en
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #11 hung while processing http://www.youronlinechoices.com/ee/opt-out-help
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #12 hung while processing
> https://www.hugedomains.com/domain_search.cfm?catSearch=437
> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #13 hung while processing
> http://hosted.ap.org/dynamic/stories/U/US_OBIT_JOHN_GLENN?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT
> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #14 hung while processing http://static.fc2.com/sh_css/common/base.css?1200605
> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #15 hung while processing https://www.hugedomains.com/terms.cfm
> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #16 hung while processing https://www.ladesk.com/comparisons/
> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #17 hung while processing http://hu.statcounter.com/features/
> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #18 hung while processing http://europa.eu/european-union/about-eu/working_el
> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #19 hung while processing http://www.atinternet.com/es/recursos/
> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
> #20 hung while processing http://ietf.org/rfc/rfc2026.txt
>
>
