It depends on the implementation you use, configured in your nutch-site.xml:
http://svn.apache.org/viewvc/nutch/trunk/src/java/org/apache/nutch/crawl/MD5Signature.java?view=markup http://svn.apache.org/viewvc/nutch/trunk/src/java/org/apache/nutch/crawl/TextProfileSignature.java?view=markup -----Original message----- > From:kiran chitturi <[email protected]> > Sent: Thu 13-Sep-2012 23:48 > To: [email protected] > Subject: nutch dedup on content of the html > > Hi, > > When crawling and indexing the document i have seen that Nutch is creating > singature and running dedup on solr which it shows as digest. > > Can anyone point how the signature is computed, is it based on the entire > text in the file ? > > Can i create signature based on only one field like 'content' so that solr > can dedup files with same content but different urls ? > > Many Thanks for your help, > -- > Kiran Chitturi >
