I guess the problem is with the package name src/java/org/apache/nutch/tools.PruneDB and src/java/org/apache/nutch/toos.PruneDB... Can you please verify again. It seems to be a typo mistake.... Thanks
keren nutch <[EMAIL PROTECTED]> wrote: Hi Matt, Thanks for reply. I put PruneDB.java in src/java/org/apache/nutch/tools and run ant. But when I run 'nutch org.apache.nutch.toos.PruneDB db -s', I got the error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/nutch/tools/PruneDB Please let me know where I'm wrong. Keren Matt Kangas wrote: I'm puzzled by the claim that "It takes ~4 hours to remove a url from the webdb.". If you're removing them one at a time, yes, because you have to rewrite the entire webdb for any change. But you want to process them in bulk. So it should only take: = (time to rewrite webdb) + (time to process 11M urls through URLFilter chain) = 4 hrs + X X depends on the complexity of your URLFilter chain. You only need RegexURLFilter with two patterns defined. (a minus for a bad site, and a plus for all else). Using my PruneDBTool, as discussed earlier, you can eliminate all of those urls in a single pass over the webdb. http://blog.busytonight.com/2006/03/nutch_07_prunedb_tool.html HTH, --Matt On Mar 22, 2006, at 12:55 PM, keren nutch wrote: > Actually, we have 11,000,000 urls in the webdb. > > Keren > > "Insurance Squared Inc." wrote: We've > got a website that is causing our crawler to slow down (from > 20mbits down to 3-5) - 400K pages that are basically not available, > we're just getting 404's. I'd like to remove them from the DB to get > our crawl speed back up again. > > Here's what our developer told me - I'm stumped, that seems really > odd. > Is there a better way to remove a URL so that it doesn't get crawled? > > Running nutch 0.71 on a dual xeon with 8 gigs of ram. > > ------------------------- > There are more than 400,000 urls in the webdb. It takes ~4 hours > to remove a url from the webdb. That means that it'll take ~1,600,000 > hours (~66,666 days, or ~2222 months, ~185 years) to remove 400,000 > CAA > urls from the webdb. Do you really want to remove them in this way? > -- Matt Kangas / [EMAIL PROTECTED] --------------------------------- Have a question? Yahoo! Canada Answers. Go to Yahoo! Canada Answers Sudhi Seshachala http://sudhilogs.blogspot.com/ --------------------------------- Yahoo! Mail Bring photos to life! New PhotoMail makes sharing a breeze.
