[ https://issues.apache.org/jira/browse/NUTCH-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108641#comment-13108641 ]
Markus Jelsma commented on NUTCH-1052: -------------------------------------- Thanks for your comments! Just to make sure i understand you correctly. You agree we should use a container object that holds the NutchDocument and the action (ADD || DELETE) and pass it to RecordWriter.write() instead of abusing NULL encoded as delete action as i do it now? Then i'd need to add a class somewehere such as: {code} class NutchIndexAction { public static enum Action ADD,DELETE; public NutchDocument doc; public Action action; } {code} And pass a NutchIndexAction instance to the record writer from IndexerMapReduce? If so, what would be the appropriate location for such a class? > Multiple deletes of the same URL using SolrClean > ------------------------------------------------ > > Key: NUTCH-1052 > URL: https://issues.apache.org/jira/browse/NUTCH-1052 > Project: Nutch > Issue Type: Improvement > Components: indexer > Affects Versions: 1.3, 1.4 > Reporter: Tim Pease > Assignee: Julien Nioche > Fix For: 1.4, 2.0 > > Attachments: NUTCH-1052-1.4-1.patch, NUTCH-1052-1.4-2.patch, > NUTCH-1052-1.4-3.patch > > > The SolrClean class does not keep track of purged URLs, it only checks the > URL status for "db_gone". When run multiple times the same list of URLs will > be deleted from Solr. For small, stable crawl databases this is not a > problem. For larger crawls this could be an issue. SolrClean will become an > expensive operation. > One solution is to add a "purged" flag in the CrawlDatum metadata. SolrClean > would then check this flag in addition to the "db_gone" status before adding > the URL to the delete list. > Another solution is to add a new state to the status field > "db_gone_and_purged". > Either way, the crawl DB will need to be updated after the Solr delete has > successfully occurred. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira