Distributed deduplication does not work right now: https://issues.apache.org/jira/browse/SOLR-3473
We've chosen not do use update processors for deduplication anymore and rely on several custom mapreduce jobs in Nutch and some custom collectors in Solr to do some on-demand online deduplication. If SOLR-3473 is fixed you can get very decent deduplication. -----Original message----- > From:Furkan KAMACI <furkankam...@gmail.com> > Sent: Thu 02-May-2013 22:30 > To: solr-user@lucene.apache.org > Subject: Pros and Cons of Using Deduplication of Solr at Huge Data Indexing > > I use Solr 4.2.1 as SolrCloud. I crawl huge data with Nutch and index them > with SolrCloud. I wonder about Solr's deduplication mechanism. What exactly > it does and does it results with a slow indexing or is it beneficial for my > situation? >