Note that I switched my two benchmarks. The 10 concurrency is the slower one.
On Fri, Jun 13, 2008 at 5:51 PM, Oleg Kalnichevski <[EMAIL PROTECTED]> wrote: > On Fri, 2008-06-13 at 17:05 +0200, Quintin Beukes wrote: >> I made some modifications to running it inside a servlet, and 51mb/s >> HttpCore, gets 30mb/s inside the servlet (while streaming it out). The >> servlet is also changed a bit, not wrapping the output stream in a >> BufferedOutputStream, and others. >> >> Either way, when the concurrency is 1 I get this 30mb/s, and the >> following summary: >> Requests per second: 5.97 [#/sec] (mean) >> Time per request: 1676.132 [ms] (mean) >> Time per request: 167.613 [ms] (mean, across all concurrent requests) >> Transfer rate: 1059.17 [Kbytes/sec] received >> >> When the concurrency is 10, I get: >> Requests per second: 29.80 [#/sec] (mean) >> Time per request: 33.557 [ms] (mean) >> Time per request: 33.557 [ms] (mean, across all concurrent requests) >> Transfer rate: 30572.62 [Kbytes/sec] received >> >> I assume this is the same problem, of connection pooling. But I'm >> using HttpCore. I googled as well as "grepped" and scan through the >> source code, but I could find connection limit preferences for >> HttpCore. Is there such a thing? >> >> Is the connection limit in the way I am using it? Do I need to manage >> the connections myself? >> > > HttpCore does _not_ provide connection management components. One can > still use org.apache.http.conn classes from the HttpClient package to > put together a lightweight HTTP client in top of HttpCore. > > Take a look at the ManagerConnectDirect sample shipped with HttpClient. > > Cheers > > Oleg > > > >> Thanks >> Q >> >> On Fri, Jun 13, 2008 at 2:53 PM, Oleg Kalnichevski <[EMAIL PROTECTED]> wrote: >> > On Fri, 2008-06-13 at 11:56 +0200, Quintin Beukes wrote: >> >> Not getting expected rate when running inside tomcat. >> >> >> >> Here are my benchmark results for running as an application (direct >> >> connections): >> >> +++++++ >> >> Benchmark 'Benchmark http clients' results. >> >> ------------------------------------------- >> >> Total time taken in milliseconds: >> >> HttpClient 4.x: 234337 >> >> HttpCore 4.x: 199655 >> >> HttpURLConnection: 209033 >> >> >> >> Total bytes transfered: 10488590000 >> >> >> >> Benchmark: HttpURLConnection >> >> Average transfer rate (bytes/second): 50176720 >> >> >> >> Benchmark: HttpCore 4.x >> >> Average transfer rate (bytes/second): 52533572 >> >> >> >> Benchmark: HttpClient 4.x >> >> Average transfer rate (bytes/second): 44758576 >> >> +++++++ >> >> >> >> Up to 52.5 MB/s using HttpCore. HttpClient being the slowest in this >> >> case. This was doing 10000 connections each downloading a 1 MB static >> >> file from Apache over a Gbit ethernet connection. >> >> >> >> Through Tomcat I was topping 8mb/s using the current HttpCore layout. >> >> But I'll let you know where the exact problem lies. I'm assuming it >> >> has something to do with the way I am writing output to tomcat's >> >> output stream. Tomcat can definitely handle super speeds as well (with >> >> static files), so I'm sure there is a way to get this all be faster. >> >> >> > >> > One important thing to watch out for is the proper sizing of the >> > HttpClient connection pool. Per default HttpClient allows only two >> > concurrent connections to the same host. So if you are hitting Tomcat >> > with 100 concurrent connections, you are very likely to end with 100 >> > Tomcat worker threads contending over 2 HttpClient connections. This >> > setup will not scale well for obvious reasons. Make sure HttpClient >> > allocates enough outbound client side connections for all inbound server >> > side connections. >> > >> > Hope this helps >> > >> > Oleg >> > >> >> If you want the above benchmark code, you can get it .class and .java >> >> in the following to jars: >> >> http://quintin.dev.junkmail.co.za/httpclient/new/benchlib.jar >> >> http://quintin.dev.junkmail.co.za/httpclient/new/benchhttp.jar >> >> >> >> I use the following command to run it (also shows the versions of >> >> httpclient/commons I'm using): >> >> java -cp >> >> benchhttp.jar:benchlib.jar:libs/commons-codec-1.3.jar:libs/commons-logging-1.0.4.jar:libs/httpclient-4.0-alpha4.jar:libs/httpcore-4.0-beta1.jar:libs/httpcore-nio-4.0-beta1.jar:libs/httpmime-4.0-alpha4.jar >> >> test.AppBench >> >> >> >> Q >> >> >> >> On Fri, Jun 13, 2008 at 11:23 AM, Oleg Kalnichevski <[EMAIL PROTECTED]> >> >> wrote: >> >> > On Fri, 2008-06-13 at 10:18 +0200, Quintin Beukes wrote: >> >> >> I'm doing ethernet. Localhost is also quite unpredictable, as it can >> >> >> go too fast. Doing ethernet I know what the limits are, ie. 100mbit or >> >> >> 1gbit. >> >> >> >> >> >> Eitherway, I setup some benchmarks and got very high troughputs. I >> >> >> also spent some time figuring out HttpCore, so it's much faster than >> >> >> all the others now. >> >> >> >> >> >> My question is then more of the sort, what could the reason for this >> >> >> be? It's probably not HttpClient's fault though. Any ideas how I can >> >> >> find the bottleneck? >> >> >> >> >> > >> >> > I am not sure I understand the problem. Is it that HttpClient tends to >> >> > be slower than HttpCore or that you are not getting the expected >> >> > throughput rate when running HttpClient inside Tomcat? >> >> > >> >> > Oleg >> >> > >> >> > >> >> > --------------------------------------------------------------------- >> >> > To unsubscribe, e-mail: [EMAIL PROTECTED] >> >> > For additional commands, e-mail: [EMAIL PROTECTED] >> >> > >> >> > >> >> >> >> >> >> >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [EMAIL PROTECTED] >> > For additional commands, e-mail: [EMAIL PROTECTED] >> > >> > >> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Quintin Beukes --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
