[
https://issues.apache.org/jira/browse/NUTCH-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Nagel updated NUTCH-2014:
-----------------------------------
Attachment: NUTCH-2014-v1.patch
The reason is a mix-up of the counters for active threads and fetch errors: the
former counter was incremented to 11 just after an error according to the logs:
{noformat}
2015-05-15 21:51:37,399 INFO fetcher.Fetcher - -activeThreads=10, ...
...
2015-05-15 21:51:38,279 INFO fetcher.FetcherThread - fetch of ... failed with:
...
...
2015-05-15 21:51:38,399 INFO fetcher.Fetcher - -activeThreads=11, ...
{noformat}
> Fetcher hang-up on completion
> -----------------------------
>
> Key: NUTCH-2014
> URL: https://issues.apache.org/jira/browse/NUTCH-2014
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.11
> Reporter: Sebastian Nagel
> Priority: Critical
> Fix For: 1.11
>
> Attachments: NUTCH-2014-v1.patch
>
>
> Although fetcher has done its work it does not shut down and exit but
> continues to log (and before reports its status to the task tracker):
> {noformat}
> -activeThreads=11, spinWaiting=0, fetchQueues.totalSize=33,
> fetchQueues.getQueueCount=1
> -activeThreads=11, spinWaiting=10, fetchQueues.totalSize=26,
> fetchQueues.getQueueCount=1
> -activeThreads=11, spinWaiting=9, fetchQueues.totalSize=9,
> fetchQueues.getQueueCount=1
> -activeThreads=9, spinWaiting=7, fetchQueues.totalSize=0,
> fetchQueues.getQueueCount=1
> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0,
> fetchQueues.getQueueCount=0
> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0,
> fetchQueues.getQueueCount=0
> ...
> (last message continues)
> {noformat}
> A possible hint: activeThreads should never exceed 10 (configured per
> default). Looks like the corresponding variable is lost/mixed-up during
> fetcher refactorization (NUTCH-1934).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
