Only thing I can think of is the -Djavax.net.debug=all or -Djavax.net.debug=ssl system properties. Debugging SSL stuff sucks. Good luck. :-(
Dan On Jan 20, 2014, at 12:31 PM, Warren, Jared S <[email protected]> wrote: > A further update: > > We had been running the performance update via HTTPS. We got some tcpdumps > taken during the test to attempt to isolate the issue to one side of the wire > (or the other). In the process, we demonstrated that our connection > pooling/keepalive was working, which made it difficult/impossible to use the > encrypted tcpdump, because we couldn't tell whether "idle" periods on a > thread were due to keepalive reuse, or were actual performance issues. > > So, we switched to using unencrypted HTTP. Viola - our performance gremlin > disappeared (we've now executed the load test 3 times to verify that we > didn't just get lucky somehow). > > So...now we're off to troubleshoot the HTTPS/SSL configuration. > > Any tips here before I dive in myself? > > > > -----Original Message----- > From: Warren, Jared S [mailto:[email protected]] > Sent: Thursday, January 16, 2014 12:22 PM > To: [email protected] > Subject: RE: CXF Client performance troubleshooting > > FWIW - we're on CXF 2.7.8. > > After cranking up some logging, we can see that the delays come between these > two log messages (note the 9 second delay between the timestamps): > > 11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG > [org.apache.cxf.transport.http.HTTPConduit] - [P14263-4866400664992057849] - > Sending POST Message with Headers to https://xxxxxxx.com/reward.asmx?wsdl > Conduit :{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit > > 11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG > [org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] - > Interceptors contributed by bus: > [com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa, > com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7, > com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a7cb0822, > com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8, > org.apache.cxf.ws.policy.PolicyInInterceptor@3166354] > > I don't know the CXF codebase well enough to interpret this (I'm downloading > source and setting up a project shortly). Can anyone help out with an > interpretation? The "com.xxx.cdi.integration.interceptor.*" are custom > interceptors written by my team. Is this an indicator they're performing > poorly? > > (we thought we had previously proved that they were performing great...but > perhaps that was incorrect). > > Thanks! > jared > > -----Original Message----- > From: Andrei Shakirin [mailto:[email protected]] > Sent: Thursday, January 16, 2014 10:55 AM > To: [email protected] > Subject: RE: CXF Client performance troubleshooting > > Hi, > > Interesting, as a guess the performance degradation in some moments can be > caused by JVM GC. I would suggest to run JVM monitor in parallel and look is > heap size will be decreased in the moments of "bad" requests. > You can try to activate CXF Logging feature (that could decrease performance > itself) or put instrumentation in own interceptors placed on the very > early/late phase of interceptor chain. > > Curious how your performance results correlates with these ones: > http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html > > Regards, > Andrei. > >> -----Original Message----- >> From: Warren, Jared S [mailto:[email protected]] >> Sent: Donnerstag, 16. Januar 2014 17:29 >> To: [email protected] >> Subject: CXF Client performance troubleshooting >> >> Hey, all - >> >> I'm facing an interesting challenge. >> >> I'm using CXF as a web services client, calling a server hosted on a >> different platform. >> >> I've instrumented my code around the call to the CXF port and observe >> that >> 10 - 20% of the requests perform very poorly (a "good" response time >> is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few >> requests that fall between "good" and "bad"). >> >> The server-side log web access log shows that all requests response >> times were "good" (100 - 200ms). For those requests that the server >> and client both think were "good", there is very little latency >> between the client measurements and the server measurements (only a >> few ms additional time in the CXF client than the server thinks the >> request took). Bandwidth on the network segments between the two is >> nowhere close to saturated. Besides that, I don't think network >> latency would inject SECONDS (perhaps milliseconds). >> >> This only occurs under very heavy load, so packet sniffing to figure >> out which side of the wire the problem is on would be extremely >> arduous (plus, I don't have easy access to sniff the relevant network >> segments). >> >> So...here's my question.... >> >> Is there any performance logging I can enable in CXF so that I can see >> "inside" >> CXF and try to push my visibility "closer to the wire"? >> >> >> >> >> >> >> >> The information transmitted is intended only for the person or entity >> to which it is addressed and may contain confidential and/or privileged >> material. >> If the reader of this message is not the intended recipient, you are >> hereby notified that your access is unauthorized, and any review, >> dissemination, distribution or copying of this message including any >> attachments is strictly prohibited. If you are not the intended >> recipient, please contact the sender and delete the material from any >> computer. > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. If the reader of this message is not the intended recipient, you > are hereby notified that your access is unauthorized, and any review, > dissemination, distribution or copying of this message including any > attachments is strictly prohibited. If you are not the intended recipient, > please contact the sender and delete the material from any computer. > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. If the reader of this message is not the intended recipient, you > are hereby notified that your access is unauthorized, and any review, > dissemination, distribution or copying of this message including any > attachments is strictly prohibited. If you are not the intended recipient, > please contact the sender and delete the material from any computer. -- Daniel Kulp [email protected] - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
