I am glad to hear that 4.0 is going to be even faster, but I was more interested in figuring out what was going on. I am actually very pleased with the performance of HttpClient over the java.net stuff. I have not tested raw stream performance, but in all other ways it is certainly far superior.
Mark -----Original Message----- From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] Sent: Thursday, November 09, 2006 12:23 PM To: HttpClient User Discussion Subject: RE: have question about buffer size On Thu, 2006-11-09 at 11:41 -0500, Mark Claassen wrote: > I just finished looking at that a bit, and I see that now that you do > read a byte at a time. Knowing this, I agree with you. > > I am now focusing on chunked input stream. I don't really know how > these things work, but it seems that the chunk size is sent every time > (?) and you read the size and then read the data. Looking at my chunk > sizes, is see that they are 8K or less. This corresponds to the behavior I am seeing. > > I must confess that I have no idea if any optimization would be > possible here. I guess you could read more than one chunk at a time, > but I don't know if that would help anything. > Mark, It is not worthwhile investing any efforts into optimizing it because HttpClient 3.x code line will no longer be actively developed past 3.1 release. HttpClient 4.0 will be based on HttpCore [1][2] which has much more memory efficient and performant buffering code among other things. Overall I expect HttpClient 4.0 be 40 to 50% faster and by an order of magnitude more memory efficient than HttpClient 3.1 due to its low lever transport code (HttpCore) [1] http://jakarta.apache.org/httpcomponents/index.html [2] http://jakarta.apache.org/httpcomponents/http-core/index.html Oleg > Mark > > -----Original Message----- > From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] > Sent: Thursday, November 09, 2006 11:22 AM > To: HttpClient User Discussion > Subject: RE: have question about buffer size > > On Thu, 2006-11-09 at 11:00 -0500, Mark Claassen wrote: > > I looked at the source a bit and it looks like the buffer size might > > be throttled. > > > > I see that my base stream is an AutoCloseInputStream, which is > > created in HttpMethodBase. > > Its source is received from HttpConnection.getResposeInputStream(). > > It looks like the input stream is actually created in open() where I > > see this code: (Version 3.01) > > > > (The sndBufSize and rcvBufSize is controlled through > > HttpConnectionParams) > > > > int outbuffersize = socket.getSendBufferSize(); > > if ((outbuffersize > 2048) || (outbuffersize <= 0)) { > > outbuffersize = 2048; > > } > > int inbuffersize = socket.getReceiveBufferSize(); > > if ((inbuffersize > 2048) || (inbuffersize <= 0)) { > > inbuffersize = 2048; > > } > > inputStream = new > > BufferedInputStream(socket.getInputStream(), > > inbuffersize); > > outputStream = new > > BufferedOutputStream(socket.getOutputStream(), outbuffersize); > > > > All this being said, even though the buffer I sent to read() is 64K > > I always seem to read 8192 bytes at a time. Looking at this code, I > > would expect to be reading only 2048 bytes at a time. I am still > > trying to figure this one out. > > > > > Mark, > > The way BufferedInputStream#read(byte[], int, int) is implemented > content looks quite intelligent to me. If there is not content in the > intermediate buffer BufferedInputStream will read directly from the > underlying input stream. So, the size of the intermediate buffer > should not matter a lot, when not reading content one byte at a time > > Oleg > > > > Mark > > > > -----Original Message----- > > From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] > > Sent: Thursday, November 09, 2006 6:11 AM > > To: HttpClient User Discussion > > Subject: Re: have question about buffer size > > > > On Wed, 2006-11-08 at 22:27 +0200, Paranoid wrote: > > > > Why should this be a problem? Over the network, maximum segments > > > > you get are (after removing framing/chunking overhead) probably > > > > that 1440 bytes in size, and httpclient is then returning you as > > > > much data as it can efficiently give at any given time. > > > > You just keep on reading all the data, piece by piece. > > > > That's how streams work; they present an abstraction over what > > > > may be (and often is) packet-based transport channel. > > > > > > some peaces are about 55 KB size, and first read is 75 KB. so why > > > are all other about 1.4 KB? > > > > There can be many factors affecting the content stream fragmentation > > and the way it is being transferred across the wire. I suggest that > > you install a traffic analyzer such as Ethereal and look at the > > packets sent by the target server. If you are absolutely convinced > > the content gets fragmented somewhere inside HttpClient, I'll dig > > into the code and try to pinpoint the problem. > > > > Oleg > > > > > and after - saving inputStream into FileOutputStream load CPU for > > > great number... this time it is really a problem... tested with > > > different buffer sized, and with greater buffer size have lower > > > CPU > load. > > > P.S.: speed of downloading about 5 MB/s... > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
