I just finished looking at that a bit, and I see that now that you do read a byte at a time. Knowing this, I agree with you.
I am now focusing on chunked input stream. I don't really know how these things work, but it seems that the chunk size is sent every time (?) and you read the size and then read the data. Looking at my chunk sizes, is see that they are 8K or less. This corresponds to the behavior I am seeing. I must confess that I have no idea if any optimization would be possible here. I guess you could read more than one chunk at a time, but I don't know if that would help anything. Mark -----Original Message----- From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] Sent: Thursday, November 09, 2006 11:22 AM To: HttpClient User Discussion Subject: RE: have question about buffer size On Thu, 2006-11-09 at 11:00 -0500, Mark Claassen wrote: > I looked at the source a bit and it looks like the buffer size might > be throttled. > > I see that my base stream is an AutoCloseInputStream, which is created > in HttpMethodBase. > Its source is received from HttpConnection.getResposeInputStream(). > It looks like the input stream is actually created in open() where I > see this code: (Version 3.01) > > (The sndBufSize and rcvBufSize is controlled through > HttpConnectionParams) > > int outbuffersize = socket.getSendBufferSize(); > if ((outbuffersize > 2048) || (outbuffersize <= 0)) { > outbuffersize = 2048; > } > int inbuffersize = socket.getReceiveBufferSize(); > if ((inbuffersize > 2048) || (inbuffersize <= 0)) { > inbuffersize = 2048; > } > inputStream = new > BufferedInputStream(socket.getInputStream(), > inbuffersize); > outputStream = new > BufferedOutputStream(socket.getOutputStream(), outbuffersize); > > All this being said, even though the buffer I sent to read() is 64K I > always seem to read 8192 bytes at a time. Looking at this code, I > would expect to be reading only 2048 bytes at a time. I am still > trying to figure this one out. > Mark, The way BufferedInputStream#read(byte[], int, int) is implemented content looks quite intelligent to me. If there is not content in the intermediate buffer BufferedInputStream will read directly from the underlying input stream. So, the size of the intermediate buffer should not matter a lot, when not reading content one byte at a time Oleg > Mark > > -----Original Message----- > From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] > Sent: Thursday, November 09, 2006 6:11 AM > To: HttpClient User Discussion > Subject: Re: have question about buffer size > > On Wed, 2006-11-08 at 22:27 +0200, Paranoid wrote: > > > Why should this be a problem? Over the network, maximum segments > > > you get are (after removing framing/chunking overhead) probably > > > that 1440 bytes in size, and httpclient is then returning you as > > > much data as it can efficiently give at any given time. > > > You just keep on reading all the data, piece by piece. > > > That's how streams work; they present an abstraction over what may > > > be (and often is) packet-based transport channel. > > > > some peaces are about 55 KB size, and first read is 75 KB. so why > > are all other about 1.4 KB? > > There can be many factors affecting the content stream fragmentation > and the way it is being transferred across the wire. I suggest that > you install a traffic analyzer such as Ethereal and look at the > packets sent by the target server. If you are absolutely convinced the > content gets fragmented somewhere inside HttpClient, I'll dig into the > code and try to pinpoint the problem. > > Oleg > > > and after - saving inputStream into FileOutputStream load CPU for > > great number... this time it is really a problem... tested with > > different buffer sized, and with greater buffer size have lower CPU load. > > P.S.: speed of downloading about 5 MB/s... > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
