[
https://issues.apache.org/jira/browse/NUTCH-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-1932:
---------------------------------
Description:
Orphan scoring filter that determines whether a page has become orphaned, e.g.
it has no more other pages linking to it. If a page hasn't been linked to after
markGoneAfter seconds, the page is marked as gone and is then removed by an
indexer. If a page hasn't been linked to after markOrphanAfter seconds, the
page is removed from the CrawlDB.
Note: if you have this plugin enabled you MUST make sure you visit 'almost'
every URL at least once within the refetch interval. If you don't, non-orphans
may be marked as orphan and get deleted! You can use NUTCH-2368 to make sure
this is the case.
was:Orphan scoring filter that determines whether a page has become orphaned,
e.g. it has no more other pages linking to it. If a page hasn't been linked to
after markGoneAfter seconds, the page is marked as gone and is then removed by
an indexer. If a page hasn't been linked to after markOrphanAfter seconds, the
page is removed from the CrawlDB.
> Automatically remove orphaned pages
> -----------------------------------
>
> Key: NUTCH-1932
> URL: https://issues.apache.org/jira/browse/NUTCH-1932
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 1.13
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.14
>
> Attachments: NUTCH-1932-add.patch, NUTCH-1932.patch,
> NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch,
> NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch,
> NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch, NUTCH-1932.patch,
> NUTCH-1932.patch, NUTCH-1932.patch
>
>
> Orphan scoring filter that determines whether a page has become orphaned,
> e.g. it has no more other pages linking to it. If a page hasn't been linked
> to after markGoneAfter seconds, the page is marked as gone and is then
> removed by an indexer. If a page hasn't been linked to after markOrphanAfter
> seconds, the page is removed from the CrawlDB.
> Note: if you have this plugin enabled you MUST make sure you visit 'almost'
> every URL at least once within the refetch interval. If you don't,
> non-orphans may be marked as orphan and get deleted! You can use NUTCH-2368
> to make sure this is the case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
