Hi Michael, > I have less than the default amount of memory allocated: > mapreduce.map.memory.mb=500; mapreduce.map.java.opts=-Xmx300m.
300 MB Java heap is probably not enough. A fetcher with 50 threads is expected keep in memory - 50 open connections with 50 buffers to hold the fetched content - the queues with URLs to fetch next - parsed and cached robots.txt rules Afaiks, also Hadoop needs a certain amount of memory to buffer the output before it's spilled to disk. If a task runs low in memory, it naturally becomes slower because CPU is used for garbage collecting. That could explain why the hang ups happen only sometimes somewhat unpredictable. Best, Sebastian On 12/16/2016 08:17 PM, Michael Coffey wrote: > Hello again, here is a lot more detail on my issue. > > I have had the same problem with many different runs, but certainly not all > of them. Typically, it seems to happen on one of several nodes of the cluster > (but not the same node each time). One time, I used numThreads=200, instead > of 50, and I got 200 hung threads at the end of the fetching task. > Nutch version is 1.12. It is running distributed on Hadoop with anywhere from > 2 to 7 slave nodes. I have no explicit timeout settings in mapred-site.xml, > nutch-site.xml, or the crawl script. I have the following fetcher.* props: > fetcher.threads.per.queue=1; fetcher.timelimit.mins=180; > fetcher.server.delay=1.0. I also have generate.max.count=25. > > I have less than the default amount of memory allocated: > mapreduce.map.memory.mb=500; mapreduce.map.java.opts=-Xmx300m. > > Looking at the logs, I don't see anything very obvious. In the 200-thread > case, the word "error" appeared only about 70 times. > > Here is an example describing 3 of the 50 hung threads of one run that > attemps to fetch about 300 resources. > 2016-12-16 04:22:45,644 WARN [main] org.apache.nutch.fetcher.Fetcher: > Aborting with 50 hung threads. > 2016-12-16 04:22:45,645 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #0 hung while processing > http://aktr.jp/news/x-girl-sports-x-aktr-02-%E3%83%97%E3%83%AD%E3%82%B8%E3%82%A7%E3%82%AF%E3%83%88%E3%82%B5%E3%82%A4%E3%83%88%E5%85%AC%E9%96%8B/ > 2016-12-16 04:22:45,647 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #1 hung while processing http://www.googletagmanager.com/ns.html?id=GTM-MF52KS > 2016-12-16 04:22:45,648 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread > #2 hung while processing http://mashable.com/2016/12/14/teens-drinking-down/ > > Earlier in the log2016-12-16 04:12:38,315 INFO [FetcherThread] > org.apache.nutch.fetcher.FetcherThread: fetching > http://aktr.jp/news/x-girl-sports-x-aktr-02-%E3%83%97%E3%83%AD%E3%82%B8%E3%82%A7%E3%82%AF%E3%83%88%E3%82%B5%E3%82%A4%E3%83%88%E5%85%AC%E9%96%8B/ > (queue crawl delay=1000ms) > > And there was no other mention of aktr.jp/news/x-girl-sports-x-aktr. However, > there were 24 akter.jp urls that it tried to fetch, 4 of which ended up on > hung threads. > > 2016-12-16 04:12:33,427 INFO [FetcherThread] > org.apache.nutch.fetcher.FetcherThread: fetching > http://www.googletagmanager.com/ns.html?id=GTM-MF52KS (queue crawl > delay=1000ms) > > And there was no other mention of googletagmanager.com. > 2016-12-16 04:12:57,991 INFO [FetcherThread] > org.apache.nutch.fetcher.FetcherThread: fetching > http://mashable.com/2016/12/14/teens-drinking-down/ (queue crawl delay=1000ms) > > And there was no other mention of mashable.com/2016/12/14/teens. There were > 17 mashable.com urls thet it tried to fetch, 2 of which ended up on hung > threads. > > I get only about 9 mentions of the word "error" in the syslog for this run. > It includes some failures to get robots.txt for various sites, some > NoHttpResponseException, and some ConnectTimeoutException. None of those > errors relate to the URLs I mentioned above > > > From: Sebastian Nagel <[email protected]> > To: [email protected] > Sent: Friday, December 9, 2016 12:42 PM > Subject: Re: Fetcher "hung while processing" > > Hi Michael, > >> What other post-fetch actions are there? > > Well, the fetched content is spilled to disk which may also become slow in > pathological cases. > > But I think it's more important to analyze what happened with the URLs > before. The logs > should contain a message "fetching ..." for every hanging URL. When does it > happen? > > If possible, let us know about > - Nutch version > - environment (local, distributed) > - configuration, esp. if not the default: > mapreduce.task.timeout > fetcher.threads.tlimeout.divisor > http.timeout > and in doubt all other modified > fetcher.* > properties > > Is the problem reproducible, or does it happen only sometimes? > > Thanks, > Sebastian > > On 12/09/2016 04:58 PM, Michael Coffey wrote: >> The property fetcher.parse is false and I pass -noParsing to the fetch >> command. What other post-fetch actions are there? >> >> >> From: Sebastian Nagel <[email protected]> >> To: [email protected] >> Sent: Friday, December 9, 2016 12:58 AM >> Subject: Re: Fetcher "hung while processing" >> >> Hi Michael, >> >> what about the property fetcher.parse ? >> >> The queue is unblocked after a page has been fetched but before parsing. >> If the parser is hanging or one of the post-fetch actions take too long >> it may happen that there are multiple URLs from the same host still in >> process. >> >> Sebastian >> >> On 12/09/2016 02:15 AM, Michael Coffey wrote: >>> I sometimes get a bunch of warning messages that say Thread #x hung while >>> processing <url> >>> Is this just a normal thing to see occasionally, or should I look to find >>> some resolution? I do have an example where the same host shows up on a >>> multitude of these messages, which puzzles me. I think there should be only >>> one thread per host, due to me specifying fetcher.threads.per.queue=1 >>> Here is example log showing the first 20 of 50 hung threads. Note that >>> http://shinystat.com and http://fabulous.com show up more than once. >>> >>> 2016-12-09 00:47:29,559 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Aborting with 50 hung threads. >>> 2016-12-09 00:47:29,560 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #0 hung while processing >>> https://www.hugedomains.com/domain_search.cfm?catSearch=434 >>> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #1 hung while processing >>> http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=820 >>> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #2 hung while processing http://shinystat.com/it/pro/info_pro.html >>> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #3 hung while processing http://events.stanford.edu/byCategory/13/ >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #4 hung while processing >>> https://www.ladesk.com/pricing/hosted/terms-and-conditions/ >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #5 hung while processing http://shinystat.com/en/opt-out_free.html >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #6 hung while processing http://shinystat.com/fr/biz/info_biz.html >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #7 hung while processing >>> http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=88 >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #8 hung while processing >>> http://www.youronlinechoices.com/sk/slovnik-pojmov >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #9 hung while processing https://twitter.com/sakura_ope >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #10 hung while processing >>> http://europa.eu/european-union/topics/culture_en >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #11 hung while processing >>> http://www.youronlinechoices.com/ee/opt-out-help >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #12 hung while processing >>> https://www.hugedomains.com/domain_search.cfm?catSearch=437 >>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #13 hung while processing >>> http://hosted.ap.org/dynamic/stories/U/US_OBIT_JOHN_GLENN?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT >>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #14 hung while processing >>> http://static.fc2.com/sh_css/common/base.css?1200605 >>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #15 hung while processing https://www.hugedomains.com/terms.cfm >>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #16 hung while processing https://www.ladesk.com/comparisons/ >>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #17 hung while processing http://hu.statcounter.com/features/ >>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #18 hung while processing >>> http://europa.eu/european-union/about-eu/working_el >>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #19 hung while processing http://www.atinternet.com/es/recursos/ >>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: >>> Thread #20 hung while processing http://ietf.org/rfc/rfc2026.txt >>> >>> >> >> >> >> >> > > > > >
