Is this possibly a dns issue. We are running a 5M page crawl and are
seeing very heavy DNS load. Just a thought.
Dennis
Stefan Neufeind wrote:
Hi,
I've encountered that here nutch is fetching quite a sum or URLs from a
long list (about 25.000). But from time to time nutch is "waiting" for
10 seconds or so. Nothing is locked, but system-load is 99,9% then. Is
nutch writing fetched data or index to disk at that stage? Is there any
way to optimize this step, e.g. by writing more often and performing the
write in "background" or caching even more in mem instead of flushing to
disk?
Regards,
Stefan
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
