[
https://issues.apache.org/jira/browse/NUTCH-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Nagel updated NUTCH-1422:
-----------------------------------
Attachment: NUTCH-1422-trunk-v1.patch
Patch for trunk: status not_modified is only determined by signature comparison
for successfully fetch documents. It does not reset the signature: the old
signature is explicitly kept for states fetch_retry and fetch_gone, it should
also for redirects. It does not harm if the not_modified detection is save. In
addition, it makes some sense for documents which disappeared temporarily to
switch again immediately after the first successful re-fetch into status
not_modified.
This solution is verified by NUTCH-1502. The issue title should be changed
accordingly, e.g., to "redirect results erroneously in status not_modified".
> reset signature for redirects
> -----------------------------
>
> Key: NUTCH-1422
> URL: https://issues.apache.org/jira/browse/NUTCH-1422
> Project: Nutch
> Issue Type: Bug
> Components: crawldb, fetcher
> Affects Versions: 1.4
> Reporter: Sebastian Nagel
> Priority: Critical
> Fix For: 1.10
>
> Attachments: NUTCH-1422-trunk-v1.patch,
> NUTCH-1422_redir_notmodified_log.txt
>
>
> In a long running continuous crawl with Nutch 1.4 URLs with a HTTP redirect
> (http.redirect.max = 0) are kept as not-modified in the CrawlDb. Short
> protocol (cf. attached dumped segment / CrawlDb data):
> 2012-02-23 : injected
> 2012-02-24 : fetched
> 2012-03-30 : re-fetched, signature changed
> 2012-04-20 : re-fetched, redirected
> 2012-04-24 : in CrawlDb as db_notmodified, still indexed with old content!
> The signature of a previously fetched document is not reset when the URL/doc
> is changed to a redirect at a later time. CrawlDbReducer.reduce then sets the
> status to db_notmodified because the new signature in with fetch status is
> identical to the old one.
> Possible fixes (??):
> * reset the signature in Fetcher
> * handle this case in CrawlDbReducer.reduce
--
This message was sent by Atlassian JIRA
(v6.2#6252)
