Do you think it is the parsing that is causing it? I was looking at a smaller fetching run and the cpu gets pushed to 100% as well but the reports keep happening. This only seems to happen when I run very large fetches (> 500K pages). I just ran a 100K fetch and it worked just fine. Should I have some special settings for larger fetches?
Dennis Andrzej Bialecki wrote: > Dennis Kubes wrote: >> Has anybody seen behavior where a fetcher duing the reduce phase will >> stop reporting and push the CPU to 100% and stay that way until the >> task times out. I am seeing this on Fedora 5 minimal running Java >> 1.5_06 on dual core processor machines with 2G of memory. I have >> tracked this down and I think this has something to do with the Java >> Inflater class. Anybody seen similar behavior? > > Could you do a kill -SIGQUIT to get a thread dump? Could you also > re-run Fetcher on the same segment (you need to delete all parts > except crawl_generate), BUT with the flag -noParsing ? > _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
