Hello Ken
I don't know, during these "hangs" there seems to be no CPU or disk
activity (the index directory keeps exactly the same size). And in this
case the site is in the LAN, so it should be quite fast to get even big
files. Before we had the fetch size limited to 64M, but now it is
unlimited and this makes no difference.
d
Ken Krugler wrote:
We ran into something that was perhaps similar with Nutch 0.7, where it
seemed like the problem was a combination of (a) really slow sites
sending us (b) really big, compressed archive files.
Our solution, which we didn't positively verify, was to limit the max
size of downloads to 10MB, and to terminate slow fetcher threads.
------------------------------------------------------------------------
Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode,
sharing):
"fetcher6" prio=1 tid=0x084c1348 nid=0x2ea5 runnable
[0x469f6000..0x469f6580]
at java.util.zip.Deflater.deflateBytes(Native Method)
at java.util.zip.Deflater.deflate(Deflater.java:284)
- locked <0x4a08c228> (a java.util.zip.Deflater)
--
Daniel Varela Santoalla
European Centre for Medium-Range Weather Forecasts (ECMWF)
(http://www.ecmwf.int)
Email: [EMAIL PROTECTED] Telephone: (+44) 118 9499608
