[
https://issues.apache.org/jira/browse/NUTCH-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-1932:
---------------------------------
Description: Orphan scoring filter that determines whether a page has
become orphaned, e.g. it has no more other pages linking to it. If a page
hasn't been linked to after markGoneAfter seconds, the page is marked as gone
and is then removed by an indexer. If a page hasn't been linked to after
markOrphanAfter seconds, the page is removed from the CrawlDB. (was: Nutch
should be able to automatically remove orphaned pages such as old 404's, and
not continue to revisit them. This requires NUTCH-1913. An inlink count of 1 is
enough.)
> Automatically remove orphaned pages
> -----------------------------------
>
> Key: NUTCH-1932
> URL: https://issues.apache.org/jira/browse/NUTCH-1932
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Attachments: NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch,
> NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch
>
>
> Orphan scoring filter that determines whether a page has become orphaned,
> e.g. it has no more other pages linking to it. If a page hasn't been linked
> to after markGoneAfter seconds, the page is marked as gone and is then
> removed by an indexer. If a page hasn't been linked to after markOrphanAfter
> seconds, the page is removed from the CrawlDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
