An easy way to do this for Nutch 0.7.1:
- Adjust regex-urlfilter.txt (as Rod mentioned), or some other
component of your URLFilter chain, to screen out the site
- Run my PruneDBTool to force all URLs in the webdb through the
URLFilter chain again
Code is here:
http://blog.busytonight.com/2006/03/nutch_07_prunedb_tool.html
(It won't work for 0.8. Hopefully won't be necessary, 'tho.)
--Matt
On Mar 17, 2006, at 1:44 PM, Insurance Squared Inc. wrote:
We've got a site that is causing our crawl to slow dramatically,
from 20mbits down to about 3 or 4. The basic problem is that the
site seems to consist of huge numbers of pages that aren't
responding. We can remove the site from the index, but it seems
like a problem to remove this site permanently from the webdb so
that we never fetch it again. Is there an easy way in 0.71 to
remove a site from the webdb, and then keep it permanently removed?
--
Matt Kangas / [EMAIL PROTECTED]
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
