To keep you updated. I haven't been able to improve my speeds above 30mb/s, but using HttpCore has proven much more powerful. Especially since the connection management is done by me, I was able to implement my other feature (smart reuse based on URL), limit concurrent connections myself, and all these.
Just figured I'd let you know. Eitherway, the real purpose of this e-mail is this. Someone mentioned I should update the wiki. I think this is a great idea, and not much effort at all. Everything I'm learning I'll document, as there doesn't seem to be anything other than examples for documentation for v4. Now, Is it this one: http://wiki.apache.org/HttpComponents/ And, what are the procedures? How should pages be named, organized, etc? Q On Fri, Jun 13, 2008 at 5:55 PM, Quintin Beukes <[EMAIL PROTECTED]> wrote: > Note that I switched my two benchmarks. The 10 concurrency is the slower one. > > On Fri, Jun 13, 2008 at 5:51 PM, Oleg Kalnichevski <[EMAIL PROTECTED]> wrote: >> On Fri, 2008-06-13 at 17:05 +0200, Quintin Beukes wrote: >>> I made some modifications to running it inside a servlet, and 51mb/s >>> HttpCore, gets 30mb/s inside the servlet (while streaming it out). The >>> servlet is also changed a bit, not wrapping the output stream in a >>> BufferedOutputStream, and others. >>> >>> Either way, when the concurrency is 1 I get this 30mb/s, and the >>> following summary: >>> Requests per second: 5.97 [#/sec] (mean) >>> Time per request: 1676.132 [ms] (mean) >>> Time per request: 167.613 [ms] (mean, across all concurrent requests) >>> Transfer rate: 1059.17 [Kbytes/sec] received >>> >>> When the concurrency is 10, I get: >>> Requests per second: 29.80 [#/sec] (mean) >>> Time per request: 33.557 [ms] (mean) >>> Time per request: 33.557 [ms] (mean, across all concurrent requests) >>> Transfer rate: 30572.62 [Kbytes/sec] received >>> >>> I assume this is the same problem, of connection pooling. But I'm >>> using HttpCore. I googled as well as "grepped" and scan through the >>> source code, but I could find connection limit preferences for >>> HttpCore. Is there such a thing? >>> >>> Is the connection limit in the way I am using it? Do I need to manage >>> the connections myself? >>> >> >> HttpCore does _not_ provide connection management components. One can >> still use org.apache.http.conn classes from the HttpClient package to >> put together a lightweight HTTP client in top of HttpCore. >> >> Take a look at the ManagerConnectDirect sample shipped with HttpClient. >> >> Cheers >> >> Oleg >> >> >> >>> Thanks >>> Q >>> >>> On Fri, Jun 13, 2008 at 2:53 PM, Oleg Kalnichevski <[EMAIL PROTECTED]> >>> wrote: >>> > On Fri, 2008-06-13 at 11:56 +0200, Quintin Beukes wrote: >>> >> Not getting expected rate when running inside tomcat. >>> >> >>> >> Here are my benchmark results for running as an application (direct >>> >> connections): >>> >> +++++++ >>> >> Benchmark 'Benchmark http clients' results. >>> >> ------------------------------------------- >>> >> Total time taken in milliseconds: >>> >> HttpClient 4.x: 234337 >>> >> HttpCore 4.x: 199655 >>> >> HttpURLConnection: 209033 >>> >> >>> >> Total bytes transfered: 10488590000 >>> >> >>> >> Benchmark: HttpURLConnection >>> >> Average transfer rate (bytes/second): 50176720 >>> >> >>> >> Benchmark: HttpCore 4.x >>> >> Average transfer rate (bytes/second): 52533572 >>> >> >>> >> Benchmark: HttpClient 4.x >>> >> Average transfer rate (bytes/second): 44758576 >>> >> +++++++ >>> >> >>> >> Up to 52.5 MB/s using HttpCore. HttpClient being the slowest in this >>> >> case. This was doing 10000 connections each downloading a 1 MB static >>> >> file from Apache over a Gbit ethernet connection. >>> >> >>> >> Through Tomcat I was topping 8mb/s using the current HttpCore layout. >>> >> But I'll let you know where the exact problem lies. I'm assuming it >>> >> has something to do with the way I am writing output to tomcat's >>> >> output stream. Tomcat can definitely handle super speeds as well (with >>> >> static files), so I'm sure there is a way to get this all be faster. >>> >> >>> > >>> > One important thing to watch out for is the proper sizing of the >>> > HttpClient connection pool. Per default HttpClient allows only two >>> > concurrent connections to the same host. So if you are hitting Tomcat >>> > with 100 concurrent connections, you are very likely to end with 100 >>> > Tomcat worker threads contending over 2 HttpClient connections. This >>> > setup will not scale well for obvious reasons. Make sure HttpClient >>> > allocates enough outbound client side connections for all inbound server >>> > side connections. >>> > >>> > Hope this helps >>> > >>> > Oleg >>> > >>> >> If you want the above benchmark code, you can get it .class and .java >>> >> in the following to jars: >>> >> http://quintin.dev.junkmail.co.za/httpclient/new/benchlib.jar >>> >> http://quintin.dev.junkmail.co.za/httpclient/new/benchhttp.jar >>> >> >>> >> I use the following command to run it (also shows the versions of >>> >> httpclient/commons I'm using): >>> >> java -cp >>> >> benchhttp.jar:benchlib.jar:libs/commons-codec-1.3.jar:libs/commons-logging-1.0.4.jar:libs/httpclient-4.0-alpha4.jar:libs/httpcore-4.0-beta1.jar:libs/httpcore-nio-4.0-beta1.jar:libs/httpmime-4.0-alpha4.jar >>> >> test.AppBench >>> >> >>> >> Q >>> >> >>> >> On Fri, Jun 13, 2008 at 11:23 AM, Oleg Kalnichevski <[EMAIL PROTECTED]> >>> >> wrote: >>> >> > On Fri, 2008-06-13 at 10:18 +0200, Quintin Beukes wrote: >>> >> >> I'm doing ethernet. Localhost is also quite unpredictable, as it can >>> >> >> go too fast. Doing ethernet I know what the limits are, ie. 100mbit or >>> >> >> 1gbit. >>> >> >> >>> >> >> Eitherway, I setup some benchmarks and got very high troughputs. I >>> >> >> also spent some time figuring out HttpCore, so it's much faster than >>> >> >> all the others now. >>> >> >> >>> >> >> My question is then more of the sort, what could the reason for this >>> >> >> be? It's probably not HttpClient's fault though. Any ideas how I can >>> >> >> find the bottleneck? >>> >> >> >>> >> > >>> >> > I am not sure I understand the problem. Is it that HttpClient tends to >>> >> > be slower than HttpCore or that you are not getting the expected >>> >> > throughput rate when running HttpClient inside Tomcat? >>> >> > >>> >> > Oleg >>> >> > >>> >> > >>> >> > --------------------------------------------------------------------- >>> >> > To unsubscribe, e-mail: [EMAIL PROTECTED] >>> >> > For additional commands, e-mail: [EMAIL PROTECTED] >>> >> > >>> >> > >>> >> >>> >> >>> >> >>> > >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: [EMAIL PROTECTED] >>> > For additional commands, e-mail: [EMAIL PROTECTED] >>> > >>> > >>> >>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> > > > > -- > Quintin Beukes > -- Quintin Beukes --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
