I'm puzzled by the claim that "It takes ~4 hours to remove a url from
the webdb.". If you're removing them one at a time, yes, because you
have to rewrite the entire webdb for any change. But you want to
process them in bulk. So it should only take:
= (time to rewrite webdb) + (time to process 11M urls through
URLFilter chain)
= 4 hrs + X
X depends on the complexity of your URLFilter chain. You only need
RegexURLFilter with two patterns defined. (a minus for a bad site,
and a plus for all else).
Using my PruneDBTool, as discussed earlier, you can eliminate all of
those urls in a single pass over the webdb.
http://blog.busytonight.com/2006/03/nutch_07_prunedb_tool.html
HTH,
--Matt
On Mar 22, 2006, at 12:55 PM, keren nutch wrote:
Actually, we have 11,000,000 urls in the webdb.
Keren
"Insurance Squared Inc." <[EMAIL PROTECTED]> wrote: We've
got a website that is causing our crawler to slow down (from
20mbits down to 3-5) - 400K pages that are basically not available,
we're just getting 404's. I'd like to remove them from the DB to get
our crawl speed back up again.
Here's what our developer told me - I'm stumped, that seems really
odd.
Is there a better way to remove a URL so that it doesn't get crawled?
Running nutch 0.71 on a dual xeon with 8 gigs of ram.
-------------------------
There are more than 400,000 urls in the webdb. It takes ~4 hours
to remove a url from the webdb. That means that it'll take ~1,600,000
hours (~66,666 days, or ~2222 months, ~185 years) to remove 400,000
CAA
urls from the webdb. Do you really want to remove them in this way?
--
Matt Kangas / [EMAIL PROTECTED]
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
