Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-23 Thread Daniel Feist
No worries. thanks! On Thu, May 22, 2014 at 1:17 PM, Oleg Kalnichevski wrote: > On Thu, 2014-05-22 at 11:05 +0100, Daniel Feist wrote: >> BTW, is there a jira issue for this I can reference? >> > > I did not raise a ticket (I have enough fun raising tickets at my day > work). You are welcome to

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-22 Thread Oleg Kalnichevski
On Thu, 2014-05-22 at 11:05 +0100, Daniel Feist wrote: > BTW, is there a jira issue for this I can reference? > I did not raise a ticket (I have enough fun raising tickets at my day work). You are welcome to raise one postmortem so to speak, if you feel like. Oleg > thanks! > > On Wed, May 2

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-22 Thread Daniel Feist
BTW, is there a jira issue for this I can reference? thanks! On Wed, May 21, 2014 at 8:30 PM, Daniel Feist wrote: >>> - 4.3 is faster than 3.1 when cpu is constrained (+5%) >>> - 4.3 is slower than 3.1 when cpu is constrained (-5%) >>> - With your changes 4.3 performs same as 3.1 on 32 core >> >

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-21 Thread Daniel Feist
>> - 4.3 is faster than 3.1 when cpu is constrained (+5%) >> - 4.3 is slower than 3.1 when cpu is constrained (-5%) >> - With your changes 4.3 performs same as 3.1 on 32 core > > This seems only to be the case with certain concurrency levels (20 and > 200), which is suspicious. Agree, weird, but a

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-21 Thread Oleg Kalnichevski
On Wed, 2014-05-21 at 16:38 +0100, Daniel Feist wrote: > You change definitely Improve things :-) unless there are other > changes in 4.3.4-SNAPSHOT that may affect performance?? > None I personally know of. > My Observations: > - Minimal is 7% faster both before and after your changes. This i

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-21 Thread Daniel Feist
You change definitely Improve things :-) unless there are other changes in 4.3.4-SNAPSHOT that may affect performance?? My Observations: - Minimal is 7% faster both before and after your changes. This is fine though, it's doing a bit more. - 4.3 is faster than 3.1 when cpu is constrained (+5%) -

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-21 Thread Oleg Kalnichevski
On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: > Here are the results!! Interesting... > > I reran everything: > - Added JVM options to even out garbage collection. > - Ran each test for minutes 5 minutes. > - Used Jetty SelectChannelConnector, same as you always used. (other > one is fa

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-21 Thread Daniel Feist
You'll see 32-core (no cpu saturation) and 4-core (cpu saturated) are in different tabs in spreadsheet.. On Wed, May 21, 2014 at 3:40 PM, Daniel Feist wrote: > Here are the results!! Interesting... > > I reran everything: > - Added JVM options to even out garbage collection. > - Ran each test fo

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-21 Thread Daniel Feist
Here are the results!! Interesting... I reran everything: - Added JVM options to even out garbage collection. - Ran each test for minutes 5 minutes. - Used Jetty SelectChannelConnector, same as you always used. (other one is faster, but less stable). https://docs.google.com/a/mulesoft.com/sprea

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-18 Thread Oleg Kalnichevski
On May 18, 2014 2:17:00 PM CEST, Oleg Kalnichevski wrote: >On Sat, 2014-05-17 at 22:10 +0100, Daniel Feist wrote: >> Sure, which jetty connector? Blocking is better because then test is >more >> about httpclient. With other connector jetty uses more cpu and has >more >> context switching. >> > >I

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-18 Thread Oleg Kalnichevski
On Sat, 2014-05-17 at 22:10 +0100, Daniel Feist wrote: > Sure, which jetty connector? Blocking is better because then test is more > about httpclient. With other connector jetty uses more cpu and has more > context switching. > I used the same micro-benchmark as before, the same jetty connector,

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-17 Thread Daniel Feist
Sure, which jetty connector? Blocking is better because then test is more about httpclient. With other connector jetty uses more cpu and has more context switching. Dan On 17 May 2014 20:15, "Oleg Kalnichevski" wrote: > On Sat, 2014-05-17 at 00:13 +0100, Daniel Feist wrote: > > > Could you pleas

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-17 Thread Oleg Kalnichevski
On Sat, 2014-05-17 at 00:13 +0100, Daniel Feist wrote: > > Could you please find out if this difference is consistent regardless of > > the CPU core number used by the system? > > Done, see other email I didn't record all figures, but did run > multiple times and records differences. > > > Honest

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-16 Thread Daniel Feist
> Could you please find out if this difference is consistent regardless of > the CPU core number used by the system? Done, see other email I didn't record all figures, but did run multiple times and records differences. > Honestly, there is really no significant differences between the two I > ca

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-16 Thread Daniel Feist
>> For some reason the HttpClient built by HttpClientBuilder, even when >> everything is turned is not only slower than minimalClient (to be >> expected) but also slower than DefaultHttpClient in previous versions. >> Not a major difference, but definitely the inverse of what of whats >> in the re

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-16 Thread Oleg Kalnichevski
On Wed, 2014-05-14 at 16:05 +0100, Daniel Feist wrote: ... > I agree 10% isn't that much and this testing maybe isn't 100% through, > but I think it's fairly clear that unless you use minimalClient with > 4.3, it doesn't matter which features you turn off (even all of them), > 4.3 still doesn't p

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-16 Thread Oleg Kalnichevski
On Thu, 2014-05-15 at 01:10 +0100, Daniel Feist wrote: > > I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > > me) HC 4.3 comfortably outperformed HC 3.1. > > Yes I see the same running this revision. Could you please find out if this difference is consistent regardless

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-16 Thread Oleg Kalnichevski
On Thu, 2014-05-15 at 01:10 +0100, Daniel Feist wrote: > > I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > > me) HC 4.3 comfortably outperformed HC 3.1. > > Yes I see the same running this revision. The difference is you are testing: > > HttpClient31 vs. HttpClients.cr

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-15 Thread Daniel Feist
> I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > me) HC 4.3 comfortably outperformed HC 3.1. > > Theories are welcome. Really don't know. Current theory is that with 32 core machine contention somehow isn't an issue. Currently re-running on dual-core machine, I'll up

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-15 Thread Daniel Feist
> I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > me) HC 4.3 comfortably outperformed HC 3.1. Yes I see the same running this revision. The difference is you are testing: HttpClient31 vs. HttpClients.createMinimal(mgr) and I was testing HttpClient31 vs. HttpClient.cu

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-14 Thread Oleg Kalnichevski
On Wed, 2014-05-14 at 11:46 +0100, Daniel Feist wrote: > Hi, > > I spent a good part of yesterday testing and comparing HttpClient > performance and the results are interesting: HttpClient 3.1 is 20% > faster than HttpClient 4.3 when both configured in the same way and > when using 50 client thre

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-14 Thread Oleg Kalnichevski
On Wed, 2014-05-14 at 11:46 +0100, Daniel Feist wrote: > Hi, > > I spent a good part of yesterday testing and comparing HttpClient > performance and the results are interesting: HttpClient 3.1 is 20% > faster than HttpClient 4.3 when both configured in the same way and > when using 50 client thre

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-14 Thread Daniel Feist
> I do not think they are configured the same way. For one, HC 4.3 uses > content decompression by default whereas HC 3.1 does not. That obviously > skews the results. Maybe but doesn't look like it, I just updated the results using 4.3 and turning off everything: HttpClients.custom().setConnecti

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-14 Thread Daniel Feist
Hi, I spent a good part of yesterday testing and comparing HttpClient performance and the results are interesting: HttpClient 3.1 is 20% faster than HttpClient 4.3 when both configured in the same way and when using 50 client threads! https://docs.google.com/a/mulesoft.com/spreadsheets/d/1Dqp2dH

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-13 Thread Oleg Kalnichevski
On Mon, 2014-05-12 at 14:10 -0700, Jaikit Savla wrote: > I have seen similar issues sometimes - while running with 512 concurrent > connections using HttpClient 4.3 (Noticed all threads stuck in > java.lang.reflect.Proxy.getProxyClass0 in jstack log). I found some > discussion online regarding

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-13 Thread Daniel Feist
Jaikit, Thats exactly what I was seeing, also with Java7. Let me do some testing though, because just because you see contention in profiler, doesn't mean that there is also a significant impact. I'll report back when I've done this. Dan On Mon, May 12, 2014 at 10:10 PM, Jaikit Savla wrote:

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-12 Thread Jaikit Savla
I have seen similar issues sometimes - while running with 512 concurrent connections using HttpClient 4.3  (Noticed all threads stuck in  java.lang.reflect.Proxy.getProxyClass0 in jstack log). I found some discussion online regarding improving the performance in latest jre but have not tried run

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-12 Thread Oleg Kalnichevski
On Mon, 2014-05-12 at 00:13 +0100, Daniel Feist wrote: > Hi, > > I'm using HttpClient in a situation where high concurrency is expected > and am doing some testing/benchmarking using gatling-tool. > > While performance isn't bad, things aren't scaling as well as I'd have > hoped. (I'm running on

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-12 Thread Oleg Kalnichevski
On Mon, 2014-05-12 at 13:21 +0100, Daniel Feist wrote: > >> 2) Can I expect 4.2 to scale better? > > > > I do not think so. In my tests HC 4.3 performs better than 4.2. There > > have also been reports > > Even with high concurrency of say 200 and high TPS, for example in a > http proxy scenario?

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-12 Thread Oleg Kalnichevski
On Mon, 2014-05-12 at 13:48 +0200, Oleg Kalnichevski wrote: > On Mon, 2014-05-12 at 00:13 +0100, Daniel Feist wrote: > > Hi, > > > > I'm using HttpClient in a situation where high concurrency is expected > > and am doing some testing/benchmarking using gatling-tool. > > > > While performance isn'

Re: Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-12 Thread Daniel Feist
>> 2) Can I expect 4.2 to scale better? > > I do not think so. In my tests HC 4.3 performs better than 4.2. There > have also been reports Even with high concurrency of say 200 and high TPS, for example in a http proxy scenario? >> 3) Is there a way of configuring 4.3.3 to not use proxys? > > No,

Thread Contention (due to use of proxys) in HttpClient 4.3

2014-05-11 Thread Daniel Feist
Hi, I'm using HttpClient in a situation where high concurrency is expected and am doing some testing/benchmarking using gatling-tool. While performance isn't bad, things aren't scaling as well as I'd have hoped. (I'm running on a amazon XL instance) When profiling with gatling running using 200