hi, I have got 30,000 urls during my crawl cycle which not unique , so i had to find out that how many are the unquie URL's as i know that "dedup" will remove the duplicate urls, but i am trying to remove the duplicate links using dedup command ,still it is not removing them can any one please help me out . The command which i had tried is below
$ bin/nutch dedup <path of index dir>/indexes thanks in advance -- View this message in context: http://www.nabble.com/dedup-is-not-removing-duplicate-record-tf3389001.html#a9434335 Sent from the Nutch - User mailing list archive at Nabble.com.
