We've confirmed that the poor performance doesn't line up with GC - that was a good theory, though.
I've been reviewing the CXF source, though, and noticed that HTTPConduit seems to support "retransmits". The behavior we see could be explained by that: somehow, the original request could be failing (for some reason) before it hits the server, so the initial request never shows up in the server logs. If we sit in a retransmit loop for some time (inside CXF) before finally reaching the server, that would explain long response times recorded in the client code, and short response times logged in the server (and, the number of requests would match). Does anyone know enough about the retransmit logic in org.apache.cxf.transport.http.HttpConduit.WrappedOutputStream enough to briefly explain it? -----Original Message----- From: Andrei Shakirin [mailto:[email protected]] Sent: Thursday, January 16, 2014 10:55 AM To: [email protected] Subject: RE: CXF Client performance troubleshooting Hi, Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests. You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain. Curious how your performance results correlates with these ones: http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html Regards, Andrei. > -----Original Message----- > From: Warren, Jared S [mailto:[email protected]] > Sent: Donnerstag, 16. Januar 2014 17:29 > To: [email protected] > Subject: CXF Client performance troubleshooting > > Hey, all - > > I'm facing an interesting challenge. > > I'm using CXF as a web services client, calling a server hosted on a > different platform. > > I've instrumented my code around the call to the CXF port and observe > that > 10 - 20% of the requests perform very poorly (a "good" response time > is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few > requests that fall between "good" and "bad"). > > The server-side log web access log shows that all requests response > times were "good" (100 - 200ms). For those requests that the server > and client both think were "good", there is very little latency > between the client measurements and the server measurements (only a > few ms additional time in the CXF client than the server thinks the > request took). Bandwidth on the network segments between the two is > nowhere close to saturated. Besides that, I don't think network > latency would inject SECONDS (perhaps milliseconds). > > This only occurs under very heavy load, so packet sniffing to figure > out which side of the wire the problem is on would be extremely > arduous (plus, I don't have easy access to sniff the relevant network > segments). > > So...here's my question.... > > Is there any performance logging I can enable in CXF so that I can see > "inside" > CXF and try to push my visibility "closer to the wire"? > > > > > > > > The information transmitted is intended only for the person or entity > to which it is addressed and may contain confidential and/or privileged > material. > If the reader of this message is not the intended recipient, you are > hereby notified that your access is unauthorized, and any review, > dissemination, distribution or copying of this message including any > attachments is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete the material from any > computer. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
