Hello again, here is a lot more detail on my issue. I have had the same problem with many different runs, but certainly not all of them. Typically, it seems to happen on one of several nodes of the cluster (but not the same node each time). One time, I used numThreads=200, instead of 50, and I got 200 hung threads at the end of the fetching task. Nutch version is 1.12. It is running distributed on Hadoop with anywhere from 2 to 7 slave nodes. I have no explicit timeout settings in mapred-site.xml, nutch-site.xml, or the crawl script. I have the following fetcher.* props: fetcher.threads.per.queue=1; fetcher.timelimit.mins=180; fetcher.server.delay=1.0. I also have generate.max.count=25.
I have less than the default amount of memory allocated: mapreduce.map.memory.mb=500; mapreduce.map.java.opts=-Xmx300m. Looking at the logs, I don't see anything very obvious. In the 200-thread case, the word "error" appeared only about 70 times. Here is an example describing 3 of the 50 hung threads of one run that attemps to fetch about 300 resources. 2016-12-16 04:22:45,644 WARN [main] org.apache.nutch.fetcher.Fetcher: Aborting with 50 hung threads. 2016-12-16 04:22:45,645 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread #0 hung while processing http://aktr.jp/news/x-girl-sports-x-aktr-02-%E3%83%97%E3%83%AD%E3%82%B8%E3%82%A7%E3%82%AF%E3%83%88%E3%82%B5%E3%82%A4%E3%83%88%E5%85%AC%E9%96%8B/ 2016-12-16 04:22:45,647 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread #1 hung while processing http://www.googletagmanager.com/ns.html?id=GTM-MF52KS 2016-12-16 04:22:45,648 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread #2 hung while processing http://mashable.com/2016/12/14/teens-drinking-down/ Earlier in the log2016-12-16 04:12:38,315 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread: fetching http://aktr.jp/news/x-girl-sports-x-aktr-02-%E3%83%97%E3%83%AD%E3%82%B8%E3%82%A7%E3%82%AF%E3%83%88%E3%82%B5%E3%82%A4%E3%83%88%E5%85%AC%E9%96%8B/ (queue crawl delay=1000ms) And there was no other mention of aktr.jp/news/x-girl-sports-x-aktr. However, there were 24 akter.jp urls that it tried to fetch, 4 of which ended up on hung threads. 2016-12-16 04:12:33,427 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread: fetching http://www.googletagmanager.com/ns.html?id=GTM-MF52KS (queue crawl delay=1000ms) And there was no other mention of googletagmanager.com. 2016-12-16 04:12:57,991 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread: fetching http://mashable.com/2016/12/14/teens-drinking-down/ (queue crawl delay=1000ms) And there was no other mention of mashable.com/2016/12/14/teens. There were 17 mashable.com urls thet it tried to fetch, 2 of which ended up on hung threads. I get only about 9 mentions of the word "error" in the syslog for this run. It includes some failures to get robots.txt for various sites, some NoHttpResponseException, and some ConnectTimeoutException. None of those errors relate to the URLs I mentioned above From: Sebastian Nagel <[email protected]> To: [email protected] Sent: Friday, December 9, 2016 12:42 PM Subject: Re: Fetcher "hung while processing" Hi Michael, > What other post-fetch actions are there? Well, the fetched content is spilled to disk which may also become slow in pathological cases. But I think it's more important to analyze what happened with the URLs before. The logs should contain a message "fetching ..." for every hanging URL. When does it happen? If possible, let us know about - Nutch version - environment (local, distributed) - configuration, esp. if not the default: mapreduce.task.timeout fetcher.threads.tlimeout.divisor http.timeout and in doubt all other modified fetcher.* properties Is the problem reproducible, or does it happen only sometimes? Thanks, Sebastian On 12/09/2016 04:58 PM, Michael Coffey wrote: > The property fetcher.parse is false and I pass -noParsing to the fetch > command. What other post-fetch actions are there? > > > From: Sebastian Nagel <[email protected]> > To: [email protected] > Sent: Friday, December 9, 2016 12:58 AM > Subject: Re: Fetcher "hung while processing" > > Hi Michael, > > what about the property fetcher.parse ? > > The queue is unblocked after a page has been fetched but before parsing. > If the parser is hanging or one of the post-fetch actions take too long > it may happen that there are multiple URLs from the same host still in > process. > > Sebastian > > On 12/09/2016 02:15 AM, Michael Coffey wrote: >> I sometimes get a bunch of warning messages that say Thread #x hung while >> processing <url> >> Is this just a normal thing to see occasionally, or should I look to find >> some resolution? I do have an example where the same host shows up on a >> multitude of these messages, which puzzles me. I think there should be only >> one thread per host, due to me specifying fetcher.threads.per.queue=1 >> Here is example log showing the first 20 of 50 hung threads. Note that >> http://shinystat.com and http://fabulous.com show up more than once. >> >> 2016-12-09 00:47:29,559 WARN [main] org.apache.nutch.fetcher.Fetcher: >> Aborting with 50 hung threads. >> 2016-12-09 00:47:29,560 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #0 hung while processing >> https://www.hugedomains.com/domain_search.cfm?catSearch=434 >> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #1 hung while processing >> http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=820 >> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #2 hung while processing http://shinystat.com/it/pro/info_pro.html >> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #3 hung while processing http://events.stanford.edu/byCategory/13/ >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #4 hung while processing >> https://www.ladesk.com/pricing/hosted/terms-and-conditions/ >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #5 hung while processing http://shinystat.com/en/opt-out_free.html >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #6 hung while processing http://shinystat.com/fr/biz/info_biz.html >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #7 hung while processing >> http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=88 >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #8 hung while processing http://www.youronlinechoices.com/sk/slovnik-pojmov >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #9 hung while processing https://twitter.com/sakura_ope >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #10 hung while processing http://europa.eu/european-union/topics/culture_en >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #11 hung while processing http://www.youronlinechoices.com/ee/opt-out-help >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #12 hung while processing >> https://www.hugedomains.com/domain_search.cfm?catSearch=437 >> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #13 hung while processing >> http://hosted.ap.org/dynamic/stories/U/US_OBIT_JOHN_GLENN?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT >> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #14 hung while processing >> http://static.fc2.com/sh_css/common/base.css?1200605 >> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #15 hung while processing https://www.hugedomains.com/terms.cfm >> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #16 hung while processing https://www.ladesk.com/comparisons/ >> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #17 hung while processing http://hu.statcounter.com/features/ >> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #18 hung while processing http://europa.eu/european-union/about-eu/working_el >> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #19 hung while processing http://www.atinternet.com/es/recursos/ >> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread >> #20 hung while processing http://ietf.org/rfc/rfc2026.txt >> >> > > > > >
