On Thu, 2006-11-09 at 11:41 -0500, Mark Claassen wrote: > I just finished looking at that a bit, and I see that now that you do read a > byte at a time. Knowing this, I agree with you. > > I am now focusing on chunked input stream. I don't really know how these > things work, but it seems that the chunk size is sent every time (?) and you > read the size and then read the data. Looking at my chunk sizes, is see > that they are 8K or less. This corresponds to the behavior I am seeing. > > I must confess that I have no idea if any optimization would be possible > here. I guess you could read more than one chunk at a time, but I don't > know if that would help anything. >
Mark, It is not worthwhile investing any efforts into optimizing it because HttpClient 3.x code line will no longer be actively developed past 3.1 release. HttpClient 4.0 will be based on HttpCore [1][2] which has much more memory efficient and performant buffering code among other things. Overall I expect HttpClient 4.0 be 40 to 50% faster and by an order of magnitude more memory efficient than HttpClient 3.1 due to its low lever transport code (HttpCore) [1] http://jakarta.apache.org/httpcomponents/index.html [2] http://jakarta.apache.org/httpcomponents/http-core/index.html Oleg > Mark > > -----Original Message----- > From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] > Sent: Thursday, November 09, 2006 11:22 AM > To: HttpClient User Discussion > Subject: RE: have question about buffer size > > On Thu, 2006-11-09 at 11:00 -0500, Mark Claassen wrote: > > I looked at the source a bit and it looks like the buffer size might > > be throttled. > > > > I see that my base stream is an AutoCloseInputStream, which is created > > in HttpMethodBase. > > Its source is received from HttpConnection.getResposeInputStream(). > > It looks like the input stream is actually created in open() where I > > see this code: (Version 3.01) > > > > (The sndBufSize and rcvBufSize is controlled through > > HttpConnectionParams) > > > > int outbuffersize = socket.getSendBufferSize(); > > if ((outbuffersize > 2048) || (outbuffersize <= 0)) { > > outbuffersize = 2048; > > } > > int inbuffersize = socket.getReceiveBufferSize(); > > if ((inbuffersize > 2048) || (inbuffersize <= 0)) { > > inbuffersize = 2048; > > } > > inputStream = new > > BufferedInputStream(socket.getInputStream(), > > inbuffersize); > > outputStream = new > > BufferedOutputStream(socket.getOutputStream(), outbuffersize); > > > > All this being said, even though the buffer I sent to read() is 64K I > > always seem to read 8192 bytes at a time. Looking at this code, I > > would expect to be reading only 2048 bytes at a time. I am still > > trying to figure this one out. > > > > > Mark, > > The way BufferedInputStream#read(byte[], int, int) is implemented content > looks quite intelligent to me. If there is not content in the intermediate > buffer BufferedInputStream will read directly from the underlying input > stream. So, the size of the intermediate buffer should not matter a lot, > when not reading content one byte at a time > > Oleg > > > > Mark > > > > -----Original Message----- > > From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] > > Sent: Thursday, November 09, 2006 6:11 AM > > To: HttpClient User Discussion > > Subject: Re: have question about buffer size > > > > On Wed, 2006-11-08 at 22:27 +0200, Paranoid wrote: > > > > Why should this be a problem? Over the network, maximum segments > > > > you get are (after removing framing/chunking overhead) probably > > > > that 1440 bytes in size, and httpclient is then returning you as > > > > much data as it can efficiently give at any given time. > > > > You just keep on reading all the data, piece by piece. > > > > That's how streams work; they present an abstraction over what may > > > > be (and often is) packet-based transport channel. > > > > > > some peaces are about 55 KB size, and first read is 75 KB. so why > > > are all other about 1.4 KB? > > > > There can be many factors affecting the content stream fragmentation > > and the way it is being transferred across the wire. I suggest that > > you install a traffic analyzer such as Ethereal and look at the > > packets sent by the target server. If you are absolutely convinced the > > content gets fragmented somewhere inside HttpClient, I'll dig into the > > code and try to pinpoint the problem. > > > > Oleg > > > > > and after - saving inputStream into FileOutputStream load CPU for > > > great number... this time it is really a problem... tested with > > > different buffer sized, and with greater buffer size have lower CPU > load. > > > P.S.: speed of downloading about 5 MB/s... > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
