Re: Counting actual input size [was: svn commit: r1088435]
On 15 April 2011 08:21, Milamber milam...@apache.org wrote: Le 14/04/2011 23:09, sebb a ecrit : On 14 April 2011 22:09, Milamber milam...@apache.org wrote: Le 14/04/2011 12:14, sebb a ecrit : On 14 April 2011 00:12, sebb seb...@gmail.com wrote: On 13 April 2011 23:33, Milamber milam...@apache.org wrote: Le 13/04/2011 14:26, sebb a ecrit : On 13 April 2011 07:53, Milamber milam...@apache.org wrote: I've updated the patch on bug 43363 since your last commit on HC4 https://issues.apache.org/bugzilla/show_bug.cgi?id=43363 With your last commit on HC4Impl, the header size and body size aren't good with a gzip stream ou chunked response. For example, with a chunked response, they are: HC4: Size in bytes: 8199 Headers size in bytes: 8192 (= Like a buffer reader?) Body size in bytes: 7 Java HC3 (good value, verified with wireshark) Size in bytes: 10505 Headers size in bytes: 581 Body size in bytes: 9924 For a gzip response: HC4: Size in bytes: 14025 (good) Headers size in bytes: 1440 Body size in bytes: 12585 Java HC3: Size in bytes: 14025 Headers size in bytes: 291 Body size in bytes: 13734 It is a bug with HttpClient 4.1 too? Possibly. Since starting to use the metrics I've found that they are mainly intended for use in custom keep-alive strategies, so may not always provide the data we want, but I'm hoping to patch HC4 to provide more useful stats in future. If you can provide details of how you are generating the test data above, I can take a further look at the problem. I've put a simple test case to show diff between plain/gzip/chunked response with the three http request type https://issues.apache.org/bugzilla/attachment.cgi?id=26885 Thanks! I can see that there is definitely a problem with the HC4 counts, and it's a bit odd. If I put a break-point just after long headerBytes = metrics.getReceivedBytesCount(); and another after long totalBytes = metrics.getReceivedBytesCount(); the headerBytes value is the same as displayed when running at full speed, but the totalBytes figure is generally much higher. Weird. I can hopefully reproduce this directly as an HC4 test case (without all the JMeter code) and use that to fix the issue. Turned out that the fix was simple (HTTPCORE-254) - the HC4 code was not always updating the metrics. That part of the code seems to depends on how much data is available, so the behaviour is timing related. I still need to fix the original issue so stats can safely be obtained for responses with no content (e.g. HEAD) (though we have a work-round for JMeter). Thanks for your works on this bug. Where download a nightly build of httpcore for test in JMeter? (or I must compile last trunk?) I've created a snapshot here: https://repository.apache.org/content/repositories/snapshots/org/apache/httpcomponents/httpcore/4.1.1-SNAPSHOT/ Thanks. I've tested, but don't works perfectly. With HC4, headers size always 1440 bytes. (total response size are good) Perhaps, we must calculate headears size like HC3 instead of use metrics? I don't see the behaviour you are reporting; HC4 seems to work OK for me with the snapshot build. Did you remove/rename the existing httpcore jar? What URL are you seeing the problem with? - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org
Re: Counting actual input size [was: svn commit: r1088435]
[snip] Thanks for your works on this bug. Where download a nightly build of httpcore for test in JMeter? (or I must compile last trunk?) I've created a snapshot here: https://repository.apache.org/content/repositories/snapshots/org/apache/httpcomponents/httpcore/4.1.1-SNAPSHOT/ Thanks. I've tested, but don't works perfectly. With HC4, headers size always 1440 bytes. (total response size are good) Perhaps, we must calculate headears size like HC3 instead of use metrics? I don't see the behaviour you are reporting; HC4 seems to work OK for me with the snapshot build. I confirm. I have this issue with test case on wiki jakarta, google home page. Sometimes with 1440 bytes ann sometimes 1338 (google, Jakarta wiki) I've found too: if the total response size (headers+body 1440 (or 1438)) HC4 have a headers size = total size, and body size is response data size (old behavior) (site with small response : www.monip.org) Did you remove/rename the existing httpcore jar? Yes. And same issue with Jdk1.5 / 6 on Linux/Windows. What URL are you seeing the problem with? - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org
Re: Counting actual input size [was: svn commit: r1088435]
On 18 April 2011 22:36, Milamber milambersp...@gmail.com wrote: [snip] Thanks for your works on this bug. Where download a nightly build of httpcore for test in JMeter? (or I must compile last trunk?) I've created a snapshot here: https://repository.apache.org/content/repositories/snapshots/org/apache/httpcomponents/httpcore/4.1.1-SNAPSHOT/ Thanks. I've tested, but don't works perfectly. With HC4, headers size always 1440 bytes. (total response size are good) Perhaps, we must calculate headears size like HC3 instead of use metrics? I don't see the behaviour you are reporting; HC4 seems to work OK for me with the snapshot build. I confirm. I have this issue with test case on wiki jakarta, google home page. Sometimes with 1440 bytes ann sometimes 1338 (google, Jakarta wiki) I've found too: if the total response size (headers+body 1440 (or 1438)) HC4 have a headers size = total size, and body size is response data size (old behavior) (site with small response : www.monip.org) I don't see the same problem as you; wiki and monip behave OK for me. Can you capture the site response using Wireshark? I wonder if there is some additional buffering going on somewhere in your ISP? Did you remove/rename the existing httpcore jar? Yes. And same issue with Jdk1.5 / 6 on Linux/Windows. What URL are you seeing the problem with? - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org
Re: Counting actual input size [was: svn commit: r1088435]
On 14 April 2011 00:12, sebb seb...@gmail.com wrote: On 13 April 2011 23:33, Milamber milam...@apache.org wrote: Le 13/04/2011 14:26, sebb a ecrit : On 13 April 2011 07:53, Milamber milam...@apache.org wrote: I've updated the patch on bug 43363 since your last commit on HC4 https://issues.apache.org/bugzilla/show_bug.cgi?id=43363 With your last commit on HC4Impl, the header size and body size aren't good with a gzip stream ou chunked response. For example, with a chunked response, they are: HC4: Size in bytes: 8199 Headers size in bytes: 8192 (= Like a buffer reader?) Body size in bytes: 7 Java HC3 (good value, verified with wireshark) Size in bytes: 10505 Headers size in bytes: 581 Body size in bytes: 9924 For a gzip response: HC4: Size in bytes: 14025 (good) Headers size in bytes: 1440 Body size in bytes: 12585 Java HC3: Size in bytes: 14025 Headers size in bytes: 291 Body size in bytes: 13734 It is a bug with HttpClient 4.1 too? Possibly. Since starting to use the metrics I've found that they are mainly intended for use in custom keep-alive strategies, so may not always provide the data we want, but I'm hoping to patch HC4 to provide more useful stats in future. If you can provide details of how you are generating the test data above, I can take a further look at the problem. I've put a simple test case to show diff between plain/gzip/chunked response with the three http request type https://issues.apache.org/bugzilla/attachment.cgi?id=26885 Thanks! I can see that there is definitely a problem with the HC4 counts, and it's a bit odd. If I put a break-point just after long headerBytes = metrics.getReceivedBytesCount(); and another after long totalBytes = metrics.getReceivedBytesCount(); the headerBytes value is the same as displayed when running at full speed, but the totalBytes figure is generally much higher. Weird. I can hopefully reproduce this directly as an HC4 test case (without all the JMeter code) and use that to fix the issue. Turned out that the fix was simple (HTTPCORE-254) - the HC4 code was not always updating the metrics. That part of the code seems to depends on how much data is available, so the behaviour is timing related. I still need to fix the original issue so stats can safely be obtained for responses with no content (e.g. HEAD) (though we have a work-round for JMeter). Thanks for all the fixes to the other code. - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org
Re: Counting actual input size [was: svn commit: r1088435]
On 14 April 2011 22:09, Milamber milam...@apache.org wrote: Le 14/04/2011 12:14, sebb a ecrit : On 14 April 2011 00:12, sebb seb...@gmail.com wrote: On 13 April 2011 23:33, Milamber milam...@apache.org wrote: Le 13/04/2011 14:26, sebb a ecrit : On 13 April 2011 07:53, Milamber milam...@apache.org wrote: I've updated the patch on bug 43363 since your last commit on HC4 https://issues.apache.org/bugzilla/show_bug.cgi?id=43363 With your last commit on HC4Impl, the header size and body size aren't good with a gzip stream ou chunked response. For example, with a chunked response, they are: HC4: Size in bytes: 8199 Headers size in bytes: 8192 (= Like a buffer reader?) Body size in bytes: 7 Java HC3 (good value, verified with wireshark) Size in bytes: 10505 Headers size in bytes: 581 Body size in bytes: 9924 For a gzip response: HC4: Size in bytes: 14025 (good) Headers size in bytes: 1440 Body size in bytes: 12585 Java HC3: Size in bytes: 14025 Headers size in bytes: 291 Body size in bytes: 13734 It is a bug with HttpClient 4.1 too? Possibly. Since starting to use the metrics I've found that they are mainly intended for use in custom keep-alive strategies, so may not always provide the data we want, but I'm hoping to patch HC4 to provide more useful stats in future. If you can provide details of how you are generating the test data above, I can take a further look at the problem. I've put a simple test case to show diff between plain/gzip/chunked response with the three http request type https://issues.apache.org/bugzilla/attachment.cgi?id=26885 Thanks! I can see that there is definitely a problem with the HC4 counts, and it's a bit odd. If I put a break-point just after long headerBytes = metrics.getReceivedBytesCount(); and another after long totalBytes = metrics.getReceivedBytesCount(); the headerBytes value is the same as displayed when running at full speed, but the totalBytes figure is generally much higher. Weird. I can hopefully reproduce this directly as an HC4 test case (without all the JMeter code) and use that to fix the issue. Turned out that the fix was simple (HTTPCORE-254) - the HC4 code was not always updating the metrics. That part of the code seems to depends on how much data is available, so the behaviour is timing related. I still need to fix the original issue so stats can safely be obtained for responses with no content (e.g. HEAD) (though we have a work-round for JMeter). Thanks for your works on this bug. Where download a nightly build of httpcore for test in JMeter? (or I must compile last trunk?) I've created a snapshot here: https://repository.apache.org/content/repositories/snapshots/org/apache/httpcomponents/httpcore/4.1.1-SNAPSHOT/ Milamber Thanks for all the fixes to the other code. - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org
Re: Counting actual input size [was: svn commit: r1088435]
On 8 April 2011 14:23, sebb seb...@gmail.com wrote: On 8 April 2011 07:45, Milamber milam...@apache.org wrote: [...] With my last submission (r1088748), I try to respond to your feedback. Please say me if another thing to improve. The problem of chunked responses still exists - such responses don't have a Content-Length header. One way round this would be to wrap the input Stream with a org.apache.commons.io.input.CountingInputStream. I don't think this will affect performance adversely. Does that make sense? Yes may be a good idea. Since your last commit on HC4Impl, entity.getContentLength() return -1 (unknown size) (but http response have a content-length define) I thinks the ResponseContentEncoding class which decompress stream is the cause. That seems likely. On HC4, I try to use a CountingInputStream on instream, but the return size is uncompressed. // InputStream instream = entity.getContent(); InputStream instream = new CountingInputStream(entity.getContent()); res.setResponseData(readResponse(res, instream, (int) entity.getContentLength())); int cnt = ((CountingInputStream) instream).getCount(); log.debug(CNT= + cnt); I thinks that CountingInputStream must be in more deep in code, directly in HttpClient, or inside the Gzip/deflate input stream? Yes, you're right - the streams we currently use are somewhat removed from the actual input . For HC3 and Java, we decompress the inputstream directly, so could wrap that with a CountingInputStream first. However, the stream contains the de-chunked data, so the chunking overhead would not be seen. But it would be closer to the true size, and might be acceptable. Ideally, one would like to intercept the input stream before de-chunking, but I'm not sure that's possible with HC3 and Java. However with HC3 and HC4 one can provide custom sockets, so it would be possible to count the actual input. One could even detect the end of the header by looking for CRLF CRLF - but that might add an unacceptable overhead, in which case we could use the current header calculation which would be reasonably close. It's not possible to provide a custom socket implementation for Java HTTP, only Java HTTPS, so this approach would not work there, so we would have to use the CountingInputStream. I suggest we use the simple approach of CountingInputStream (CIS) for Java and HC3; it's easy to do and fairly accurate. No point spending lots of time on those implementations as HC4 is better. I'll have a look at HC4 to see what can be done - would you like to see if CIS works OK for HC3 and Java? HC4 keeps metrics on the connection, so it's very easy to find the actual byte counts for header and body. == I think we should consider changing the default to be the total network response size. However, this may affect some size assertions. HTTPSampleResult (v2.4) stores the decoded body size only. Maybe we should store the header and raw body sizes separately, rather than combining some of them. This would give the most flexibility. Also, consider adding the fields to SampleResult rather than HTTPSampleResult. For non-HTTP responses, the headerSize would normally be zero and raw body size would be the same as decoded body size, but e.g. for POP3 perhaps it would make sense to implement header size. Adding the fields to SampleResult would also make it easier to display them in the Tree View Listener (HTTPSampleResult is currently defined in a different jar which is built later - perhaps that's a mistake). - To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org
