I'm using the mapred branch on a FreeBSD 7.0 box to do fetchs of a
300k url list.
Initially, its able to reach ~25 pages/s with 150 threads. The
fetcher gets progressivly slower though, dropping down to about
~15 pages/s after about 2-3 hours or so and continues to slow
down. I've seen a few references on these lists to the issue, but
I'm not clear on if its expected behaviour or if there's a
solution to it? I've also noticed that the process takes up more
and more memory as it runs, is this expected as well?
We've run into a similar situation, though we're using Nutch 0.7.
What seems to be happening is that a host is slowly trickling data
back to us. This happens when we're trying to releasing the
connection, and we get stuck in the commons-httpclient code at
ChunkedInputStream.exhaustInputStream().
I have a theory that this happens when our http protocol max size
limit is hit. The protocol-httpclient plugin reads up to the limit
(in our case, 1MB) and then tries to release the connection, but
for some reason the host keeps sending us data, albeit at some very
slow rate. I was seeing 30Kbits/second or so.
Anyway, I've added the commons-httpclient code to my project and am
plugging in some additional logging to help track down the issue.
I would appreciate any feedback. Please also note that you need to
eliminate other factors, like the limit of threads per host, but
most notably the overhead of parsing - please use the -noParse flag
to fetcher for all those experiments. In the past it was common for
the fetcher to be stuck in a buggy parser plugin, so you will need
to eliminate this factor.
We're only using the html & text parsers, so I don't think that's the
problem. Plus we dumping the thread stack when it hangs, and it's
always in the ChunkedInputStream.exhaustInputStream() process (see
trace below).
We've left the # of threads per host set to 1, and varied the total
number of threads from 50 up to 400. Increasing from 50 to 200
definitely improved performance, but going from 200 to 400 seemed to
have minimal impact, other than boosting the CPU usage to 80%.
More research results to come...
-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-470-9200
