I'm crawling a local server. I have Nutch 2 working on a local machine with the default 1G heap size. I got several OOM errors, but the fetch eventually finishes.
In order to get rid of the OOM errors, I moved everything to a machine with more memory and increased the heap size to 8G. However, I'm still getting the OOM errors and now I'm having Nutch abort hung threads. After it aborts the hung threads, Nutch itself hangs. Any idea what could be causing this or what to look at? hadoop.log shows nothing after the "Aborting with 1 hung threads." message. Thanks.
