Marc Layer created CXF-6674:
-------------------------------
Summary: Infinite loop when handling SOAP responses in
split/parallel Camel route
Key: CXF-6674
URL: https://issues.apache.org/jira/browse/CXF-6674
Project: CXF
Issue Type: Bug
Components: JAX-WS Runtime
Affects Versions: 2.7.10
Environment: Apache Camel
2.12.3
Apache CXF
2.7.10
App Server
WebSphere Platform 8.0.0.10 [ND 8.0.0.10 cf101502.03]
Java Runtime Version
pxa6460_26sr8fp2ifx-20141114_01 (SR8 FP2)
Java Vendor
IBM Corporation
Java Version
1.6.0
Java Compiler
j9jit26
Java VM name
IBM J9 VM
Operating System
Linux
OS Architecture
x86_64
OS Version
2.6.32-279.el6.x86_64
VM Information
JRE 1.6.0 Linux amd64-64 Compressed References 20141114_222464 (JIT enabled,
AOT enabled)
J9VM - R26_Java626_SR8_20141114_1023_B222464
JIT - r11.b07_20141003_74578.01
GC - R26_Java626_SR8_20141114_1023_B222464_CMPRSS
J9CL - 20141114_222464
VM Vendor
IBM Corporation
VM Version
2.6
Reporter: Marc Layer
We are using Apache Camel 2.12.3 and Apache CXF 2.7.10 to integrate some
backend SOAP web services. In our production environment we are facing the
issue that at irregular intervals a CXF worker thread would hang in a busy
loop, consuming 100% CPU time. The only recovery so far is to restart the
server.
The loop seems to occur in
org.apache.cxf.jaxws.context.WrappedMessageContext.copyScoped():
private void copyScoped(Map<String, Object> msg) {
for (String s : scopes.keySet()) {
message.put(s, msg.get(s));
}
}
I guess it might be related to a threading issue that causes the iterator of
scopes.keySet() to loop infinitely. Using the VM monitoring tool Dynatrace, I
can see that the iterated "scope" map is an unsynchronized HashMap. I am unsure
where the "scope" map was created and how it could have leaked into another
thread. "message" seems to be an instance of MessageImpl, which is also derived
from HashMap and therefore unsynchronized. I attached a Dynatrace screenshot
from one such invocation.
Multi-threading in our case is two-fold:
- CXF seems to be using separate worker threads even for synchronous
invocations.
- Additionally the call in question makes use of Camel's splitter EIP with
parallel processing which also uses separate threads (RouteBuilder is
attached), effectively like this:
.split(body(), dataAggregator)
.stopOnException()
.parallelProcessing().executorServiceRef("multicastExecutorService")
.to("cxf:bean:backendService")
I analyzed a number of cases in which two web service calls were dispatched at
the same time. Dynatrace shows that the backend web services always finished
within about 80-130ms.
Unfortunately we were unable to reproduce the error on a testing system where
we could have attached a remote debugger to the application, not even under
heavy load. It only occured in the production environment so far. Therefore I
am uncertain if the problem only occurs when Camel's parallelProcessing() is
used. We will disable this feature in the application's next release (end of
next week) to verify this.
There was another issue, CXF-6160 "AsyncResponse infinite loops", in which an
infinite loop in MessageImpl was tackled by making ExchangeImpl a subclass of
ConcurrentHashMap rather than a subclass of StringImpl (which is also derived
from HashMap). Maybe these issues are related. The fix, however, was
implemented only for CXF 3.x. Unfortunately we are bound to using Java 1.6 as
Websphere 8 is only available for IBM JRE 1.6. The latest Camel version
available for Java 1.6 comes with CXF 2.7.14, later Camel versions are only
available for Java 1.7. I think we could easily upgrade to a newer 2.7.x
version of CXF but not to CXF 3.x.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
