[
https://issues.apache.org/jira/browse/CXF-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marc Layer updated CXF-6674:
----------------------------
Attachment: BackendServiceRoute.java
dynatrace.png
> Infinite loop when handling SOAP responses in split/parallel Camel route
> ------------------------------------------------------------------------
>
> Key: CXF-6674
> URL: https://issues.apache.org/jira/browse/CXF-6674
> Project: CXF
> Issue Type: Bug
> Components: JAX-WS Runtime
> Affects Versions: 2.7.10
> Environment: Apache Camel
> 2.12.3
> Apache CXF
> 2.7.10
> App Server
> WebSphere Platform 8.0.0.10 [ND 8.0.0.10 cf101502.03]
> Java Runtime Version
> pxa6460_26sr8fp2ifx-20141114_01 (SR8 FP2)
> Java Vendor
> IBM Corporation
> Java Version
> 1.6.0
> Java Compiler
> j9jit26
> Java VM name
> IBM J9 VM
> Operating System
> Linux
> OS Architecture
> x86_64
> OS Version
> 2.6.32-279.el6.x86_64
> VM Information
> JRE 1.6.0 Linux amd64-64 Compressed References 20141114_222464 (JIT enabled,
> AOT enabled)
> J9VM - R26_Java626_SR8_20141114_1023_B222464
> JIT - r11.b07_20141003_74578.01
> GC - R26_Java626_SR8_20141114_1023_B222464_CMPRSS
> J9CL - 20141114_222464
> VM Vendor
> IBM Corporation
> VM Version
> 2.6
> Reporter: Marc Layer
> Attachments: BackendServiceRoute.java, dynatrace.png
>
>
> We are using Apache Camel 2.12.3 and Apache CXF 2.7.10 to integrate some
> backend SOAP web services. In our production environment we are facing the
> issue that at irregular intervals a CXF worker thread would hang in a busy
> loop, consuming 100% CPU time. The only recovery so far is to restart the
> server.
> The loop seems to occur in
> org.apache.cxf.jaxws.context.WrappedMessageContext.copyScoped():
> private void copyScoped(Map<String, Object> msg) {
> for (String s : scopes.keySet()) {
> message.put(s, msg.get(s));
> }
> }
> I guess it might be related to a threading issue that causes the iterator of
> scopes.keySet() to loop infinitely. Using the VM monitoring tool Dynatrace, I
> can see that the iterated "scope" map is an unsynchronized HashMap. I am
> unsure where the "scope" map was created and how it could have leaked into
> another thread. "message" seems to be an instance of MessageImpl, which is
> also derived from HashMap and therefore unsynchronized. I attached a
> Dynatrace screenshot from one such invocation.
> Multi-threading in our case is two-fold:
> - CXF seems to be using separate worker threads even for synchronous
> invocations.
> - Additionally the call in question makes use of Camel's splitter EIP with
> parallel processing which also uses separate threads (RouteBuilder is
> attached), effectively like this:
> .split(body(), dataAggregator)
> .stopOnException()
>
> .parallelProcessing().executorServiceRef("multicastExecutorService")
> .to("cxf:bean:backendService")
> I analyzed a number of cases in which two web service calls were dispatched
> at the same time. Dynatrace shows that the backend web services always
> finished within about 80-130ms.
> Unfortunately we were unable to reproduce the error on a testing system where
> we could have attached a remote debugger to the application, not even under
> heavy load. It only occured in the production environment so far. Therefore I
> am uncertain if the problem only occurs when Camel's parallelProcessing() is
> used. We will disable this feature in the application's next release (end of
> next week) to verify this.
> There was another issue, CXF-6160 "AsyncResponse infinite loops", in which an
> infinite loop in MessageImpl was tackled by making ExchangeImpl a subclass of
> ConcurrentHashMap rather than a subclass of StringImpl (which is also derived
> from HashMap). Maybe these issues are related. The fix, however, was
> implemented only for CXF 3.x. Unfortunately we are bound to using Java 1.6 as
> Websphere 8 is only available for IBM JRE 1.6. The latest Camel version
> available for Java 1.6 comes with CXF 2.7.14, later Camel versions are only
> available for Java 1.7. I think we could easily upgrade to a newer 2.7.x
> version of CXF but not to CXF 3.x.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
