Hi Dennis, thank you for the answer. Hmm, could theoretically be. But to prevent this the server already does resolving completely on his local machine. Also I wonder about the CPU-load moving to "system" - I suspected heavy disk-access or so ... but I don't know how/when the fetcher writes data to disk etc.
Regards, Stefan Dennis Kubes wrote: > Is this possibly a dns issue. We are running a 5M page crawl and are > seeing very heavy DNS load. Just a thought. > > Dennis > > Stefan Neufeind wrote: >> Hi, >> >> I've encountered that here nutch is fetching quite a sum or URLs from a >> long list (about 25.000). But from time to time nutch is "waiting" for >> 10 seconds or so. Nothing is locked, but system-load is 99,9% then. Is >> nutch writing fetched data or index to disk at that stage? Is there any >> way to optimize this step, e.g. by writing more often and performing the >> write in "background" or caching even more in mem instead of flushing to >> disk? ------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
