Hi Steven, > is the ordering of dedup and index wrong No, that's correct: it would be not really efficient to first index duplicates and then remove them afterwards.
If I understand right the db_gone pages have previously been indexed (and were successfully fetched), right? > but "bin/nutch dedup" removes the records entirely A dedup job should neither remove records entirely, they are only set to status db_duplicate, nor should it touch anything except db_fetched and db_notmodified. If it does that's a bug. Can you send the exact commands of "nutch dedup" and "nutch index"? Have you checked the crawldb before and after using "bin/nutch readdb" to get some hints what's special with these urls or documents? Thanks, Sebastian On 07/03/2015 11:37 AM, Hayles, Steven wrote: > I'm using bin/crawl on Nutch 1.9 (with Solr 4.10.3) > > What I see is that "bin/nutch update" sets db_gone status correctly, but > "bin/nutch dedup" removes the records entirely before "bin/nutch index" can > tell Sol to remove them from its index. > > Is dedup doing more than it should, is the ordering of dedup and index wrong, > or is there some configuration that I have wrong? > > Thanks > > Steven Hayles > Systems Analyst > > IT Services, University of Leicester, > Propsect House, 94 Regent Rd, Leicester, LE1 7DA, UK > > T: +44 (0)116 229 7950 > E: [email protected]<mailto:[email protected]> > > The Queen's Anniversary Prizes 1994, 2002 & 2013 > THE Awards Winners 2007-2013 > > Elite without being elitist > > Follow us on Twitter http://twitter.com/uniofleicester or > visit our Facebook page https://facebook.com/UniofLeicester > >
