Hello, I am trying to do a breadth first crawl using nutch. Currently, the generator lets me specify the max # of URLs to be generated for fetching. If the # of unfetched URLs in the current crawldb is too large, I would like to create multiple fetchlists using one generate task, by specifying a batchSize parameter. For e.g., topN=10M and batchSize=1M will create at most 10 segments each 1M URLs in its fetchlist (the last segment could have less). I went through the discussion here: http://issues.apache.org/jira/browse/NUTCH-171. I have to agree with Rod and Stefan here that a larger job is much more likely to fail than several smaller jobs. That is why I was wondering if I can modify the generator to create multiple output segments somehow. Any idea as to how this can be accomplished? Thank you, Vishal.
