A further update:
We had been running the performance update via HTTPS. We got some tcpdumps taken during
the test to attempt to isolate the issue to one side of the wire (or the other). In the
process, we demonstrated that our connection pooling/keepalive was working, which made it
difficult/impossible to use the encrypted tcpdump, because we couldn't tell whether
"idle" periods on a thread were due to keepalive reuse, or were actual
performance issues.
So, we switched to using unencrypted HTTP. Viola - our performance gremlin
disappeared (we've now executed the load test 3 times to verify that we didn't
just get lucky somehow).
So...now we're off to troubleshoot the HTTPS/SSL configuration.
Any tips here before I dive in myself?
-----Original Message-----
From: Warren, Jared S [mailto:[email protected]]
Sent: Thursday, January 16, 2014 12:22 PM
To: [email protected]
Subject: RE: CXF Client performance troubleshooting
FWIW - we're on CXF 2.7.8.
After cranking up some logging, we can see that the delays come between these
two log messages (note the 9 second delay between the timestamps):
11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG
[org.apache.cxf.transport.http.HTTPConduit] -
[P14263-4866400664992057849] - Sending POST Message with Headers to
https://xxxxxxx.com/reward.asmx?wsdl Conduit
:{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit
11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG
[org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] -
Interceptors contributed by bus:
[com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa,
com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7
,
com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a
7cb0822,
com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8
, org.apache.cxf.ws.policy.PolicyInInterceptor@3166354]
I don't know the CXF codebase well enough to interpret this (I'm downloading source and
setting up a project shortly). Can anyone help out with an interpretation? The
"com.xxx.cdi.integration.interceptor.*" are custom interceptors written by my
team. Is this an indicator they're performing poorly?
(we thought we had previously proved that they were performing great...but
perhaps that was incorrect).
Thanks!
jared
-----Original Message-----
From: Andrei Shakirin [mailto:[email protected]]
Sent: Thursday, January 16, 2014 10:55 AM
To: [email protected]
Subject: RE: CXF Client performance troubleshooting
Hi,
Interesting, as a guess the performance degradation in some moments can be caused by JVM
GC. I would suggest to run JVM monitor in parallel and look is heap size will be
decreased in the moments of "bad" requests.
You can try to activate CXF Logging feature (that could decrease performance
itself) or put instrumentation in own interceptors placed on the very
early/late phase of interceptor chain.
Curious how your performance results correlates with these ones:
http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-pl
atform.html
Regards,
Andrei.
-----Original Message-----
From: Warren, Jared S [mailto:[email protected]]
Sent: Donnerstag, 16. Januar 2014 17:29
To: [email protected]
Subject: CXF Client performance troubleshooting
Hey, all -
I'm facing an interesting challenge.
I'm using CXF as a web services client, calling a server hosted on a
different platform.
I've instrumented my code around the call to the CXF port and observe
that
10 - 20% of the requests perform very poorly (a "good" response time
is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few
requests that fall between "good" and "bad").
The server-side log web access log shows that all requests response
times were "good" (100 - 200ms). For those requests that the server
and client both think were "good", there is very little latency
between the client measurements and the server measurements (only a
few ms additional time in the CXF client than the server thinks the
request took). Bandwidth on the network segments between the two is
nowhere close to saturated. Besides that, I don't think network
latency would inject SECONDS (perhaps milliseconds).
This only occurs under very heavy load, so packet sniffing to figure
out which side of the wire the problem is on would be extremely
arduous (plus, I don't have easy access to sniff the relevant network segments).
So...here's my question....
Is there any performance logging I can enable in CXF so that I can see "inside"
CXF and try to push my visibility "closer to the wire"?
The information transmitted is intended only for the person or entity
to which it is addressed and may contain confidential and/or privileged
material.
If the reader of this message is not the intended recipient, you are
hereby notified that your access is unauthorized, and any review,
dissemination, distribution or copying of this message including any
attachments is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which
it is addressed and may contain confidential and/or privileged material. If the
reader of this message is not the intended recipient, you are hereby notified
that your access is unauthorized, and any review, dissemination, distribution
or copying of this message including any attachments is strictly prohibited. If
you are not the intended recipient, please contact the sender and delete the
material from any computer.
The information transmitted is intended only for the person or entity to which
it is addressed and may contain confidential and/or privileged material. If the
reader of this message is not the intended recipient, you are hereby notified
that your access is unauthorized, and any review, dissemination, distribution
or copying of this message including any attachments is strictly prohibited. If
you are not the intended recipient, please contact the sender and delete the
material from any computer.
