Is this possibly a dns issue. We are running a 5M page crawl and are
seeing very heavy DNS load. Just a thought.
Dennis
Stefan Neufeind wrote:
Hi,
I've encountered that here nutch is fetching quite a sum or URLs from a
long list (about 25.000). But from time to time nutch is "waiting" for
10 seconds or so. Nothing is locked, but system-load is 99,9% then. Is
nutch writing fetched data or index to disk at that stage? Is there any
way to optimize this step, e.g. by writing more often and performing the
write in "background" or caching even more in mem instead of flushing to
disk?
Regards,
Stefan
