Re: Dedup Question

[Dennis Kubes]

It will remove the one with the lowest score in the crawldb as set by 
the scoring filters.  Dedup first removes by url then by content hash. 
If the content is changed even slightly though it will *not* be detected 
as a duplicate.  Solving that problem is called near duplicate detection 
(ndd) and uses an algorithm called shingling which isn't currently 
implemented in Nutch (but hopefully will be in the near future).

Dennis

Patrick Markiewicz wrote:
Hi,

            If I have a url http://www.example.com/index.html stored in
my index with the content: EMPTY FILE, and I have a file
http://www.domain.com/index.html with the content: EMPTY FILE, then the
two files are duplicates.  Which one will the de-duplication process
remove from the index?  Thanks.

 

Patrick