Hi Dennis, thank you for the answer. Hmm, could theoretically be. But to prevent this the server already does resolving completely on his local machine. Also I wonder about the CPU-load moving to "system" - I suspected heavy disk-access or so ... but I don't know how/when the fetcher writes data to disk etc.
Regards, Stefan Dennis Kubes wrote: > Is this possibly a dns issue. We are running a 5M page crawl and are > seeing very heavy DNS load. Just a thought. > > Dennis > > Stefan Neufeind wrote: >> Hi, >> >> I've encountered that here nutch is fetching quite a sum or URLs from a >> long list (about 25.000). But from time to time nutch is "waiting" for >> 10 seconds or so. Nothing is locked, but system-load is 99,9% then. Is >> nutch writing fetched data or index to disk at that stage? Is there any >> way to optimize this step, e.g. by writing more often and performing the >> write in "background" or caching even more in mem instead of flushing to >> disk?
