A further update: We had been running the performance update via HTTPS. We got some tcpdumps taken during the test to attempt to isolate the issue to one side of the wire (or the other). In the process, we demonstrated that our connection pooling/keepalive was working, which made it difficult/impossible to use the encrypted tcpdump, because we couldn't tell whether "idle" periods on a thread were due to keepalive reuse, or were actual performance issues.
So, we switched to using unencrypted HTTP. Viola - our performance gremlin disappeared (we've now executed the load test 3 times to verify that we didn't just get lucky somehow). So...now we're off to troubleshoot the HTTPS/SSL configuration. Any tips here before I dive in myself? -----Original Message----- From: Warren, Jared S [mailto:[email protected]] Sent: Thursday, January 16, 2014 12:22 PM To: [email protected] Subject: RE: CXF Client performance troubleshooting FWIW - we're on CXF 2.7.8. After cranking up some logging, we can see that the delays come between these two log messages (note the 9 second delay between the timestamps): 11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG [org.apache.cxf.transport.http.HTTPConduit] - [P14263-4866400664992057849] - Sending POST Message with Headers to https://xxxxxxx.com/reward.asmx?wsdl Conduit :{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit 11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG [org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] - Interceptors contributed by bus: [com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa, com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7, com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a7cb0822, com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8, org.apache.cxf.ws.policy.PolicyInInterceptor@3166354] I don't know the CXF codebase well enough to interpret this (I'm downloading source and setting up a project shortly). Can anyone help out with an interpretation? The "com.xxx.cdi.integration.interceptor.*" are custom interceptors written by my team. Is this an indicator they're performing poorly? (we thought we had previously proved that they were performing great...but perhaps that was incorrect). Thanks! jared -----Original Message----- From: Andrei Shakirin [mailto:[email protected]] Sent: Thursday, January 16, 2014 10:55 AM To: [email protected] Subject: RE: CXF Client performance troubleshooting Hi, Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests. You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain. Curious how your performance results correlates with these ones: http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html Regards, Andrei. > -----Original Message----- > From: Warren, Jared S [mailto:[email protected]] > Sent: Donnerstag, 16. Januar 2014 17:29 > To: [email protected] > Subject: CXF Client performance troubleshooting > > Hey, all - > > I'm facing an interesting challenge. > > I'm using CXF as a web services client, calling a server hosted on a > different platform. > > I've instrumented my code around the call to the CXF port and observe > that > 10 - 20% of the requests perform very poorly (a "good" response time > is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few > requests that fall between "good" and "bad"). > > The server-side log web access log shows that all requests response > times were "good" (100 - 200ms). For those requests that the server > and client both think were "good", there is very little latency > between the client measurements and the server measurements (only a > few ms additional time in the CXF client than the server thinks the > request took). Bandwidth on the network segments between the two is > nowhere close to saturated. Besides that, I don't think network > latency would inject SECONDS (perhaps milliseconds). > > This only occurs under very heavy load, so packet sniffing to figure > out which side of the wire the problem is on would be extremely > arduous (plus, I don't have easy access to sniff the relevant network > segments). > > So...here's my question.... > > Is there any performance logging I can enable in CXF so that I can see > "inside" > CXF and try to push my visibility "closer to the wire"? > > > > > > > > The information transmitted is intended only for the person or entity > to which it is addressed and may contain confidential and/or privileged > material. > If the reader of this message is not the intended recipient, you are > hereby notified that your access is unauthorized, and any review, > dissemination, distribution or copying of this message including any > attachments is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete the material from any > computer. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
