Edward Quick wrote:
Dennis Kubes wrote:
If you are using more than one index then dedup will not work across
indexes.
This is incorrect. DeleteDuplicates works just fine with multiple
indexes, assuming you process all indexes in the same run of
DeleteDuplicates, so that it has a global view of all input indexes.
A single index should dedup correctly unless the pages are not
exact duplicates but near duplicates. The dedup process works on url
and byte hash. If the content is even 1 byte different, it doesn't work.
This depends on the implementation of Signature. Indeed, the default
MD5HashSignature works this way.
Near duplicate detection is another set of algorithms that haven't been
implemented in Nutch yet.
Well, the existing TextProfileSignature can be used as a form of (crude)
near-duplicate detection, precisely because it is tolerant to small
changes in the input text.
Thanks Andrzej.
How do you tell Nutch to use the TextProfileSignature instead of
MD5HashSignature for deduplicating?
See the following property in your nutch-site.xml: db.signature.class.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
