Hi Michael, what about the property fetcher.parse ?
The queue is unblocked after a page has been fetched but before parsing. If the parser is hanging or one of the post-fetch actions take too long it may happen that there are multiple URLs from the same host still in process. Sebastian On 12/09/2016 02:15 AM, Michael Coffey wrote: > I sometimes get a bunch of warning messages that say Thread #x hung while > processing <url> > Is this just a normal thing to see occasionally, or should I look to find > some resolution? I do have an example where the same host shows up on a > multitude of these messages, which puzzles me. I think there should be only > one thread per host, due to me specifying fetcher.threads.per.queue=1 > Here is example log showing the first 20 of 50 hung threads. Note that > http://shinystat.com and http://fabulous.com show up more than once. > > 2016-12-09 00:47:29,559 WARN [main] org.apache.nutch.fetcher.Fetcher: > Aborting with 50 hung threads. > 2016-12-09 00:47:29,560 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #0 hung while processing > https://www.hugedomains.com/domain_search.cfm?catSearch=434 > 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #1 hung while processing > http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=820 > 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #2 hung while processing http://shinystat.com/it/pro/info_pro.html > 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #3 hung while processing http://events.stanford.edu/byCategory/13/ > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #4 hung while processing > https://www.ladesk.com/pricing/hosted/terms-and-conditions/ > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #5 hung while processing http://shinystat.com/en/opt-out_free.html > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #6 hung while processing http://shinystat.com/fr/biz/info_biz.html > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #7 hung while processing > http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=88 > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #8 hung while processing http://www.youronlinechoices.com/sk/slovnik-pojmov > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #9 hung while processing https://twitter.com/sakura_ope > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #10 hung while processing http://europa.eu/european-union/topics/culture_en > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #11 hung while processing http://www.youronlinechoices.com/ee/opt-out-help > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #12 hung while processing > https://www.hugedomains.com/domain_search.cfm?catSearch=437 > 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #13 hung while processing > http://hosted.ap.org/dynamic/stories/U/US_OBIT_JOHN_GLENN?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT > 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #14 hung while processing http://static.fc2.com/sh_css/common/base.css?1200605 > 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #15 hung while processing https://www.hugedomains.com/terms.cfm > 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #16 hung while processing https://www.ladesk.com/comparisons/ > 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #17 hung while processing http://hu.statcounter.com/features/ > 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #18 hung while processing http://europa.eu/european-union/about-eu/working_el > 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #19 hung while processing http://www.atinternet.com/es/recursos/ > 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #20 hung while processing http://ietf.org/rfc/rfc2026.txt > >
