Hello Ken I don't know, during these "hangs" there seems to be no CPU or disk activity (the index directory keeps exactly the same size). And in this case the site is in the LAN, so it should be quite fast to get even big files. Before we had the fetch size limited to 64M, but now it is unlimited and this makes no difference.
d Ken Krugler wrote: > > We ran into something that was perhaps similar with Nutch 0.7, where it > seemed like the problem was a combination of (a) really slow sites > sending us (b) really big, compressed archive files. > > Our solution, which we didn't positively verify, was to limit the max > size of downloads to 10MB, and to terminate slow fetcher threads. >> ------------------------------------------------------------------------ >> >> Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode, >> sharing): >> >> "fetcher6" prio=1 tid=0x084c1348 nid=0x2ea5 runnable >> [0x469f6000..0x469f6580] >> at java.util.zip.Deflater.deflateBytes(Native Method) >> at java.util.zip.Deflater.deflate(Deflater.java:284) >> - locked <0x4a08c228> (a java.util.zip.Deflater) -- Daniel Varela Santoalla European Centre for Medium-Range Weather Forecasts (ECMWF) (http://www.ecmwf.int) Email: [EMAIL PROTECTED] Telephone: (+44) 118 9499608 Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
