Hi,
Some of my searches return duplicate pages, so I wanted to remove these. I'm
not exactly sure how to do this but tried the command below, and restarted
Tomcat, but still got the same results.
I'm using the Nutch-Nightly from about 2 weeks ago. Just wondered if I'm
doing something wrong here?
Thanks.
$ nutch dedup -local -workingdir /www/nutch/planetbp
/www/nutch/planetbp/segments
run java in /usr/j2sdk1.4.2_03
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611171518/index(/www/nutch/planetbp/segments/20050611171518/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611171700/index(/www/nutch/planetbp/segments/20050611171700/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611173224/index(/www/nutch/planetbp/segments/20050611173224/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611181430/index(/www/nutch/planetbp/segments/20050611181430/index)
050612 092326 Clearing old deletions in
/www/nutch/planetbp/segments/20050611184455/index(/www/nutch/planetbp/segments/20050611184455/index)
050612 092327 Clearing old deletions in
/www/nutch/planetbp/segments/20050611185714/index(/www/nutch/planetbp/segments/20050611185714/index)
050612 092327 Clearing old deletions in
/www/nutch/planetbp/segments/20050611190051/index(/www/nutch/planetbp/segments/20050611190051/index)
050612 092327 Clearing old deletions in
/www/nutch/planetbp/segments/20050611190155/index(/www/nutch/planetbp/segments/20050611190155/index)
050612 092327 Reading url hashes...
050612 092328 parsing file:/www/nutch-nightly/conf/nutch-default.xml
050612 092329 parsing file:/www/nutch-nightly/conf/nutch-site.xml
050612 092330 Sorting url hashes...
050612 092331 Deleting url duplicates...
050612 092331 Deleted 0 url duplicates.
050612 092331 Reading content hashes...
050612 092331 Sorting content hashes...
050612 092331 Deleting content duplicates...
050612 092331 Deleted 309 content duplicates.
050612 092332 Duplicate deletion complete locally. Now returning to NFS...
050612 092332 DeleteDuplicates complete
$
-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Nutch-general mailing list
Nutch-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-general
