Hi, Evidently, there is some work to be done here. The good thing is that there is a comprehenisve amount of correspondence on all of the issues quoted above. Maybe we can clean the Jira up a bit.
On Wed, Nov 9, 2011 at 1:42 AM, Markus Jelsma <[email protected]>wrote: > Thanks! > > > Before opening a ticket, searched the JIRA and found that this is the > same > > problem as reported in NUTCH-1089, NUTCH-990 and NUTCH-1112. I just found > > it via different symptoms. NUTCH-1089 offers a patch. > > > > > -----Original Message----- > > > From: Markus Jelsma [mailto:[email protected]] > > > Sent: Tuesday, 8 November 2011 7:19 PM > > > To: [email protected] > > > Subject: Re: A bug has been fixed in protocol-httpclient > > > > > > Hi, > > > > > > Can you open a Jira ticket and attach a patch file so we can track it? > > > > > > Thanks > > > > > > > Hi guys, > > > > > > > > I know that protocol-httpclient is not recommended to use because of > > > > > > known > > > > > > > problems, but I don't have much choice because I need authentication > > > > support, as a few other people do as well, I am sure. > > > > > > > > I've reported a problem with too aggressive de-duplication recently. > > > > > > On the > > > > > > > example that I had, I traced that problem to an empty content field. > > > > Digging further, I found this in httpclient/HttpResponse.java (lines > > > > > > > > 126-130): > > > > while ((bufferFilled = in.read(buffer, 0, buffer.length)) != > > > > > > -1 > > > > > > > && totalRead + bufferFilled < contentLength) { > > > > > > > > totalRead += bufferFilled; > > > > out.write(buffer, 0, bufferFilled); > > > > > > > > } > > > > > > > > This should be changed to > > > > > > > > while ( ( bufferFilled = in.read( buffer, 0, buffer.length ) > > > > > > ) != > > > > > > > -1 ) { > > > > > > > > int toWrite = totalRead + bufferFilled < contentLength ? > > > > > > > > totalRead + > > > > > > bufferFilled : > > > > contentLength - totalRead ; totalRead += bufferFilled; > > > > > > > > out.write( buffer, 0, toWrite ) ; > > > > if ( totalRead >= contentLength ) break ; > > > > > > > > } > > > > > > > > Else the last read portion quite often is not stored. Obviously, this > > > > > > is > > > > > > > causing problems, especially in small documents where the last read > > > > portion is the only one, and in PDF documents, as well as other > > > > > > document > > > > > > > types that are sensitive to truncation. > > > > > > > > This problem explains a large part of false de-duplication cases, as > > > > > > well > > > > > > > as parsing errors with truncated content symptoms, but it does not > > > > > > seem to > > > > > > > explain all of them. > > > > > > > > Regards, > > > > > > > > Arkadi > -- *Lewis*
