The default GC which I think is still parallel. But I should explicitly set it. I’m working on getting JMX monitors up so that I can track ActiveMQ counters but also GC state.
I can’t get a reliable and short term failure to occur so right now I’ve focused on mitigating the issue and easily rebuilding my queue when it crashes. On Tue, Dec 23, 2014 at 8:55 AM, Tim Bain <tb...@alumni.duke.edu> wrote: > BTW, you never did answer the question about which GC strategy you're > using, and it occurred to me that if you're using CMS, lots of full GCs > that don't actually reclaim much memory after a long time being up is the > classic failure scenario for CMS. It happens when Old Gen gets fragmented, > which in turn happens because CMS is a non-compacting GC strategy in Old > Gen. If you're using CMS and seeing continual full GCs, you should look at > whether G1 GC would be better for your needs. > > On Mon, Dec 22, 2014 at 11:03 PM, Kevin Burton <bur...@spinn3r.com> wrote: > > > Great feedback. Thanks btw. I’m working on getting up better JMX > monitors > > so I can track memory here more aggressively. Bumping up memory by 1.5G > > temporarily fixed the problem. However, it seems correlated to the number > > of connections. So I suspect I’ll just hit this again in the next few > > weeks. > > > > By that time I plan to have better JMX monitors in place to resolve this. > > > > On Sat, Dec 20, 2014 at 10:28 PM, Tim Bain <tb...@alumni.duke.edu> > wrote: > > > > > > What JVM are you using, and what GC strategy with which options? And > for > > > that matter, what broker version? > > > > > > With Hotspot 7u21 and G1GC while running a long-running performance > > stress > > > test I've observed that Old Gen use increases over time (despite the > fact > > > that G1GC is supposed to collect Old Gen during its normal collection > > > operations), and GCs against Old Gen happen semi-continually after Old > > Gen > > > hits a certain memory threshold. However, unlike what you're > observing, > > 1) > > > the GCs I saw were Old Gen GCs but not full GCs (G1 allows GCing Old > Gen > > > during incremental GCs), 2) the broker remains responsive with > reasonable > > > pause times close to my target, and 3) once Old Gen hits the 90% > > threshold > > > that forces a full GC, that full GC is able to successfully collect > > nearly > > > all of the Old Gen memory. My conclusion from that was that although > > > objects were being promoted to Old Gen (and I tried unsuccessfully to > > > prevent that from occurring, see > > > > > > > > > http://activemq.2283324.n4.nabble.com/Potential-Bug-in-Master-Slave-with-Replicated-LevelDB-Store-td4686450.html > > > ), > > > nearly all of them were unreachable by the time a full GC actually > > > occurred. > > > > > > So if you're seeing continual full GCs (not just Old Gen GCs if you're > > > using G1) that don't actually free any Old Gen memory, then what you're > > > seeing different behavior than I saw, and it means that the objects in > > Old > > > Gen are still reachable. One possible reason for that would be > messages > > > still being held in destinations waiting to be consumed; look for > queues > > > without consumers (especially DLQs), as well as durable subscribers > that > > > are offline. If you're certain that's not the case, maybe you can post > > > some of the results of analyzing the heap snapshot so that people who > > know > > > the codebase better could see if anything jumps out? > > > > > > On Sat, Dec 20, 2014 at 1:51 PM, Kevin Burton <bur...@spinn3r.com> > > wrote: > > > > > > > I’m trying to diagnose a long term memory leak with ActiveMQ. > > > > > > > > Basically , my app runs fine for about a week or so, then goes to > 100% > > > CPU > > > > doing continually full GCs back to back. > > > > > > > > No work is done during that period. > > > > > > > > I have a large number of sessions to the AMQ box, but things are fine > > on > > > > startup. > > > > > > > > It’s entirely possible that y app isn’t releasing resources, but I”m > > > trying > > > > to figure out the best way to track that down. > > > > > > > > I’m using org.apache.activemq.UseDedicatedTaskRunner=false so that > > thread > > > > pools are used. Which apparently can cause a bit of wasted memory. > > > > > > > > I have a heap snapshot. I loaded that into the Eclipse Memory > Analyzer > > > and > > > > didn’t see any obvious candidates but of course I’m not an expert on > > the > > > > ActiveMQ code base. > > > > > > > > Are there any solid JMX counters I can track during this process? > > Number > > > > of sessions? etc. > > > > > > > > -- > > > > > > > > Founder/CEO Spinn3r.com > > > > Location: *San Francisco, CA* > > > > blog: http://burtonator.wordpress.com > > > > … or check out my Google+ profile > > > > <https://plus.google.com/102718274791889610666/posts> > > > > <http://spinn3r.com> > > > > > > > > > > > > > -- > > > > Founder/CEO Spinn3r.com > > Location: *San Francisco, CA* > > blog: http://burtonator.wordpress.com > > … or check out my Google+ profile > > <https://plus.google.com/102718274791889610666/posts> > > <http://spinn3r.com> > > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
