[
https://issues.apache.org/jira/browse/NUTCH-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463104#comment-17463104
]
Sebastian Nagel commented on NUTCH-2912:
----------------------------------------
Looks good! I've added the new FetchCompleteCrawlDatumProcessor to the property
hostdb.crawldatum.processors, created a hostdb from a crawldb: the
"perc.complete" numbers are correct. [~markus17], please go ahead and commit!
Maybe one caveat for documentation: the unfetched status includes fetches
failed with transient errors until {{db.fetch.retry.max}} is reached. If there
are transient errors, it may take some time (3 days using the default
configuration) until 100% completeness are reached.
> CrawlDatumProcessor to calculate crawl completeness
> ---------------------------------------------------
>
> Key: NUTCH-2912
> URL: https://issues.apache.org/jira/browse/NUTCH-2912
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Major
> Attachments: NUTCH-2912-1.patch, NUTCH-2912.patch
>
>
> Simple processor that calculates the completeness of the crawl per host.
>
> This does not account for known unknowns, e.g. unfetched URLs that haven't
> been found yet. Therefore, the calculated percentage can be highly volatile
> for beginning crawls.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
