Hi, we are experiencing a lot of OutOfMemory errors in productions under heavy load with servicemix 3.3.2, cxf 2.2.6, jdk1.5, linux.
We use the a Cxf client generated from java annotations. Analysing the problem with a profiler I found out that many messages (NormalizedMessageImpl) are retained inside the JaxWsClientProxy responseContext. The messages are for many different operations and all have status DONE. The number of messages does not grow linearly with the requests. Sometimes a single request adds 6 messages, sometimes none. Sometime 100 requests add 4 messages, sometimes 10. Looking at the code I noticed that the the responseContext is cleared only just before a new invocation. It looks like all the last context is retained if no new requests are done. The component use a cxf-se servicemix pool with 16 to 200 threads. My hypothesis is that if a "dirty thread" is picked up the old context is cleared, a new almost identical one is added and I notice no increase in instance count. If another thread is chosen I see up to 6 new "lost" messages. When the load grows and the number of threads does the same the number of "lost contexts" grows all threads eventually became "dirty" and we have an OOM. What do you think? If this is the actual scenario, what can I do to fix this? Has already been fixed in a more recent release? I'd prefer not to upgrade cxf becaus the 2.2.6 is the standard release for servicemix 3.3.2. With some dirty reflection tricks I could reach the "lost context" and clear it after each request. I'm going to try this right after this mail to see if it solves. Are there any other better options? Thanks, bye Lorenzo
