[
https://issues.apache.org/jira/browse/NUTCH-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187716#comment-13187716
]
Sebastian Nagel commented on NUTCH-1247:
----------------------------------------
Interestingly, I also found a couple of URLs with unreasonable high retry
counter in the data where NUTCH-1245 was first observed (it was Nutch 1.2).
* all these URLs failed with some exception (invalid URI or HTTP=403) and not
404, not found, or robots denied?
Markus, do the URLs which overflow the retry counter in your Db also belong
to this class?
* in the segments the status of these URLs is fetch_retry (in crawl_fetch):
In Fetcher.java the case ProtocolStatus.EXCEPTION inside the switch statement
in FetcherThread.run() falls through the default where the result is collected
with STATUS_FETCH_RETRY.
CrawlDbReducer calls FetchSchedule.forceRefetch() only for the cases
STATUS_FETCH_NOT_MODIFIED or STATUS_FETCH_GONE (here via setPageGoneSchedule).
The branch STATUS_FETCH_RETRY does not reset the retry counter. Generator never
calls forceRefetch() nor does it reset the retry counter.
If this analysis is correct there are two possible patches:
* A (CrawlDbReducer): call setPageGoneSchedule for the case STATUS_FETCH_RETRY
* B (Generator): reset the retry counter to zero when a db_gone URL is
generated again
> CrawlDatum.retries should be int
> --------------------------------
>
> Key: NUTCH-1247
> URL: https://issues.apache.org/jira/browse/NUTCH-1247
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Fix For: 1.5
>
>
> CrawlDatum.retries is a byte and goes bad with larger values.
> 12/01/12 18:35:22 INFO crawl.CrawlDbReader: retry -127: 1
> 12/01/12 18:35:22 INFO crawl.CrawlDbReader: retry -128: 1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
