Hi,
Some of my searches return duplicate pages, so I wanted to remove these. I'm
not exactly sure how to do this but tried the command below, and restarted
Tomcat, but still got the same results.
I'm using the Nutch-Nightly from about 2 weeks ago. Just wondered if I'm
doing something wrong here?
Thanks.
$ nutch dedup -local -workingdir /www/nutch/planetbp
/www/nutch/planetbp/segments
run java in /usr/j2sdk1.4.2_03
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611171518/index(/www/nutch/planetbp/segments/20050611171518/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611171700/index(/www/nutch/planetbp/segments/20050611171700/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611173224/index(/www/nutch/planetbp/segments/20050611173224/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611181430/index(/www/nutch/planetbp/segments/20050611181430/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611184455/index(/www/nutch/planetbp/segments/20050611184455/index)
050612 092327 Clearing old deletions in
/www/nutch/planetbp/segments/20050611185714/index(/www/nutch/planetbp/segments/20050611185714/index)
050612 092327 Clearing old deletions in
/www/nutch/planetbp/segments/20050611190051/index(/www/nutch/planetbp/segments/20050611190051/index)
050612 092327 Clearing old deletions in
/www/nutch/planetbp/segments/20050611190155/index(/www/nutch/planetbp/segments/20050611190155/index)
050612 092327 Reading url hashes...
050612 092328 parsing file:/www/nutch-nightly/conf/nutch-default.xml
050612 092329 parsing file:/www/nutch-nightly/conf/nutch-site.xml
050612 092330 Sorting url hashes...
050612 092331 Deleting url duplicates...
050612 092331 Deleted 0 url duplicates.
050612 092331 Reading content hashes...
050612 092331 Sorting content hashes...
050612 092331 Deleting content duplicates...
050612 092331 Deleted 309 content duplicates.
050612 092332 Duplicate deletion complete locally. Now returning to NFS...
050612 092332 DeleteDuplicates complete
$
