Hi Freeman, I can't believe my eyes, but I no longer have the memory problems now. I've never seen our services actually slowly use less and less memory, rather than climb in memory usage and do erratic garbage collections. The thing that fixed it was using the new servicemix-jms endpoints. As soon as I switched to the jms:consumer and the jms:provider endpoints, my memory usage problems COMPLETELY went away. I just can't stop staring at my jmx console since I've been fighting this problem a long time. I had tried to switch to the new jms endpoints a long time ago but ran into a lot of problems. So for anyone interest, I'll outline the couple things that caused me to end up so screwed up for a long time, especially if some notes could be made in the documentation that could help others:
1. Using the new jms:consumer endpoint will hang if you don't set the DONE status properly on an exchange you receive from it (such as in a servicemix-bean component). 2. Using the old jms:endpoint, you will get an exception if you DO set the DONE status properly on an exchange you receive from it (you actually have to not set the DONE status, or it will cause an error). 3. Using the new jms:provider endpoint, if you have multiple JMS provider endpoints, all but one of them will hang on start up if you configure them in separate service units and follow the instructions on this page: http://servicemix.apache.org/servicemix-jms-new-endpoints.html#servicemix-jmsnewendpoints-SimpleExamples - Particularly the problem is with the 2 lines if tells you to use for the jms:provider: <property name="maxConnections" value="1" /> <property name="maximumActive" value="1" /> If you use those two lines and have multiple JMS provider endpoints (in separate service units, not sure about if they are in the same service unit), then all but one of them will lock up on the borrowObject method in the activemq pool. Apparently, those settings only allow one connection total between all the consumers and it starves the other consumers. I'm pretty sure my assessments are correct above, if not, then feel free to correct me, but as I fixed one problem, such as setting the DONE status, I would get a new problem causing me to revert the change. Now that I've handled the issues above, I'm on the new JMS endpoints and my memory usage is now completely stabilized and remaining consistent while it's been rocessing 1 exchange every second for the last 40 minutes. Ryan On Fri, Dec 26, 2008 at 11:37 AM, Ryan Moquin <[email protected]>wrote: > I'm definitely setting the done status correctly, or at least now I am. I > noticed that if you set the DONE status when receiving from the old JMS > endpoint, you get an exception. If you set the DONE status in your > servicemix-bean when receiving from the new jms endpoint, then it's fine. I > think that issue with the old endpoint is what had me so screwed up with my > servicemix-bean. Anyhow, as for the memory issue, when I see the garbage > collection in jconsole, it seems to garbage all but a small amount of memory > each time a message comes into one of my services and finishes processing. > This will continue where the memory usage will slowly creep up. I'll see it > commit an extra meg periodically to the JVM as the memory peaks near the > current allocated heap size. If I hit the garbage collection button, it > will garbage collect all but the small amount of memory that seems to be > creeping up as each message processes. If I hit the GC button a few times > it appears that it ends up garbage collecting that slow amount of > accumulating memory. I'm a little puzzled as to why it doesn't seem like it > garbage collect all the dead objects, but I guess maybe it's just something > in the JVM and maybe I shouldn't worry about it. I still don't like that it > commits more memory when it shouldn't have to. So I will just assume that > it's not leaking memory since after hitting the GC button a few times it > seems like it will eventually free the accumulating memory. > > On Tue, Dec 23, 2008 at 10:11 PM, Freeman Fang <[email protected]>wrote: > >> Ryan Moquin wrote: >> >>> I wanted to do a little probing regarding memory and servicemix. I >>> was wondering how exchanges between services are stored/persisted. >>> I'm asking because I've been plagued by out of memory exceptions with >>> my heap memory. Under high load, servicemix can exhaust 1 gig of >>> memory in 3 hours. >>> >>> I finally have been integration testing servicemix services with junit >>> (which is awesome to be able to do by the way), when I run my junit >>> perf test through jprob, I see my memory keep steadily climbing. When >>> I look at the results, it shows the onMessageExchange method call of >>> my servicemix-bean component as being the place that uses the most >>> memory. All my services temporary objects get disposed of, but it >>> looks like the MessageExchanges are hanging around. It particulary >>> points to InOnly and NormalizedMessage as being the biggest offenders. >>> >>> >>> >> Is it possible that your bean didn't set DONE status correctly and send ME >> with DONE back? >> >> Is servicemix keeping all the exchanges in memory, whether finished or >>> not? Does my analysis sound correct? Is there any tweaking I need to >>> do out of the box to have exchanges purged? I want to make sure I'm >>> correct on the issue and understand what's going on enough to be able >>> to fix it. >>> >>> Thanks, ryan >>> >>> >>> >> >> >
