[
https://issues.apache.org/jira/browse/NUTCH-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108641#comment-13108641
]
Markus Jelsma commented on NUTCH-1052:
--------------------------------------
Thanks for your comments! Just to make sure i understand you correctly. You
agree we should use a container object that holds the NutchDocument and the
action (ADD || DELETE) and pass it to RecordWriter.write() instead of abusing
NULL encoded as delete action as i do it now?
Then i'd need to add a class somewehere such as:
{code}
class NutchIndexAction {
public static enum Action ADD,DELETE;
public NutchDocument doc;
public Action action;
}
{code}
And pass a NutchIndexAction instance to the record writer from
IndexerMapReduce? If so, what would be the appropriate location for such a
class?
> Multiple deletes of the same URL using SolrClean
> ------------------------------------------------
>
> Key: NUTCH-1052
> URL: https://issues.apache.org/jira/browse/NUTCH-1052
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Affects Versions: 1.3, 1.4
> Reporter: Tim Pease
> Assignee: Julien Nioche
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1052-1.4-1.patch, NUTCH-1052-1.4-2.patch,
> NUTCH-1052-1.4-3.patch
>
>
> The SolrClean class does not keep track of purged URLs, it only checks the
> URL status for "db_gone". When run multiple times the same list of URLs will
> be deleted from Solr. For small, stable crawl databases this is not a
> problem. For larger crawls this could be an issue. SolrClean will become an
> expensive operation.
> One solution is to add a "purged" flag in the CrawlDatum metadata. SolrClean
> would then check this flag in addition to the "db_gone" status before adding
> the URL to the delete list.
> Another solution is to add a new state to the status field
> "db_gone_and_purged".
> Either way, the crawl DB will need to be updated after the Solr delete has
> successfully occurred.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
