> > I did some research and I traced the problem to be somewhere inside
> > HttpRequest of protocol-httpclient.
> I had a similar report from someone else, and I'll try to find out what
> is happening. Thanks for this debugging output, it is helpful - if you
> find something else, please let me know.
It seems, that at least in most cases (dunno if in every case) inside
the HttpResponse, in the line
while ((bufferFilled = in.read(buffer, 0, buffer.length)) != -1 &&
tryAndRead > 0) {
read returns just one byte (bufferFilled == 1). Normally it returns
buffer.length, and it also returns full buffers from the same socket,
but for some reason it goes rampage
and starts returning one byte at a time.
I created an ugly workaround by creating a counter, which starts from 10
and degreases every time when bufferFilled == 1. Once the counter
reaches zero, it aborts the read by breaking the inner while loop. This
makes the fetched page to be corrupted, but at least it won't halt
the whole fetch of thousands pages.
- Juho Mäkinen, http://www.juhonkoti.net
