Shailesh Kochhar wrote:
I not very familiar with the Nutch API though I know there's a MD5 signature based deduping method in place and a Signature class to extend for offline duplicate detection. I was wondering if anyone had tried search time deduping and what would be good places to try and implement it.
Nutch already does search-time deduping. By default it limits things to two hits per host, but you can dedup by other fields and with other per-dup counts. This is available through NutchBean:
http://lucene.apache.org/nutch/apidocs/org/apache/nutch/searcher/NutchBean.html#search(org.apache.nutch.searcher.Query,%20int,%20int,%20java.lang.String) and though the OpenSearch servlet. Doug
