[ http://issues.apache.org/jira/browse/NUTCH-371?page=comments#action_12440092 ] Andrzej Bialecki commented on NUTCH-371: -----------------------------------------
Thanks for investigating this. Regarding the updated version (please create diffs in the future): wouldn't it be easier to make createKey() an abstract method, which TestInputFormat and MD5HashInputFormat override, and then just use if (key instanceof MD5Hash) in RecordReader? > DeleteDuplicates should remove documents with duplicate URLs > ------------------------------------------------------------ > > Key: NUTCH-371 > URL: http://issues.apache.org/jira/browse/NUTCH-371 > Project: Nutch > Issue Type: Bug > Components: indexer > Reporter: Chris Schneider > Attachments: DeleteDuplicates2.java, patch.txt > > > DeleteDuplicates is supposed to delete documents with duplicate URLs (after > deleting documents with identical MD5 hashes), but this part is apparently > not yet implemented. Here's the comment from DeleteDuplicates.java: > // 2. map indexes -> <<url, fetchdate>, <index,doc>> > // partition by url > // reduce, deleting all but most recent. > // > // Part 2 is not yet implemented, but the Indexer currently only indexes one > // URL per page, so this is not a critical problem. > It is apparently also known that re-fetching the same URL (e.g., one month > later) will result in more than one document with the same URL (this is > alluded to in NUTCH-95), but the comment above suggests that the indexer will > solve the problem before DeleteDuplicates, because it will only index one > document per URL. > This is not necessarily the case if the segments are to be divided among > search servers, as each server will have its own index built from its own > portion of the segments. Thus, if the URL in question was fetched in > different segments, and these segments end up assigned to different search > servers, then the indexer can't be relied on to eliminate the duplicates. > Thus, it seems like the second part of the DeleteDuplicates algorithm (i.e., > deleting documents with duplicate URLs) needs to be implemented. I agree with > Byron and Andrzej that the most recently fetched document (rather than the > one with the highest score) should be preserved. > Finally, it's also possible to get duplicate URLs in the segments without > re-fetching an expired URL in the crawldb. This can happen if 3 different > URLs all redirect to the target URL. This is yet another consequence of > handling redirections immediately, rather than adding the target URL to the > crawldb for fetching in some subsequent segment (see NUTCH-273). -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
