Is it possible to purge low-scoring urls from the crawldb? My news crawl has many thousands of zero-scoring urls and also many thousands of urls with scores less than 0.03. These urls will never be fetched because they will never make it into the generator's topN by score. So, all they do is make the process slower.
It seems like something an urlfilter could do, but I have not found any documentation for any urlfilter that does it.
