I was able to execute a crawl of couple of hundred thousand URLs in local mode , I did not get any OOM exceptions , what machine configuration do you use ?
On Sat, Nov 30, 2013 at 4:43 PM, Amit Sela <am...@infolinks.com> wrote: > I get OOM exception in parse phase. > I think it's related to https://issues.apache.org/jira/browse/NUTCH-1640 > Did anyone succeed in fetching and parsing hundreds of thousands or even > millions of pages with Nutch 1.7 ? >
