It's not a JVM GC, but executing the broker GC (which you can execute via JMX) which flushes buffers, etc. And it's flushing of the cursor's prefetch buffer that does the trick.
The main annoying thing about this problem is that I cannot reproduce it. And it only happens in our production environment once in a while. I fear it might be a race condition in the JRE6 we use. So upgrading to JRE7 might solve the problem. On 3 September 2015 at 17:57, Tim Bain <tb...@alumni.duke.edu> wrote: > For me, the weirdest part about this is the fact that a full GC somehow > fixes is. GCs aren't supposed to do anything that would affect the > application (garbage should always be dead, so there should be no effect to > anything that's still alive), so that's really strange. > > If you start ActiveMQ with a debug port enabled, you can always grab the > 5.4.2 source code and attach a debugger once you get into this situation > and see if stepping through the code gives any clues to how you got there > (and whether the code that "should have" returned values from method calls > really is)... > > And I hope you're planning that upgrade for as soon as you can arrange it. > > Tim > > On Wednesday, August 19, 2015, Michiel Hendriks < > michiel.hendr...@mp-objects.com> wrote: > > > We've got a really weird problem. Some times we still have messages on a > > queue, but they are not being consumed by the active consumers. > > > > Via JMX the QueueSize is 1 > > doesCursorHaveMessagesBuffered return false > > cursorSize returns 0 > > doesCursorHaveSpace returns true > > I can browse the message > > > > Restarting consumers has no effect. Restarting ActiveMQ, or simply > > performing a gc on the broker via JMX will correct it. Sending a message > to > > the queue will push the current pending message to a consumer, but the > new > > message is then stuck. > > > > I've created and analyzed a memory dump. The batchList > (OrderedPendingList) > > of the QueueStorePrefetch contains a map with size 1, so > > doesCursorHaveMessagesBuffered should have returned true rather than > false. > > > > We have been unable to create a reproduction case yet, it just happens at > > some point at stays like that until a restart or explicit gc on the > broker. > > The queue has 20 consumers (prefetch=1), and a lot of messages do need to > > be redelivered. > > > > This is with ActiveMQ 5.4.2 running on java 1.6.0_36 > > (rhel-1.13.8.1.el5_11-x86_64). Yes, I know it's rather old (we don't have > > the luxury yet to upgrade). Looking at the code of a newer ActiveMQ I do > > not see differences in the path from doesCursorHaveMessagesBuffered to > > OrderedPendingList.isEmpty > > > > Has anybody seen anything like this before? > > > > I did see a message from Donnell Alwyn on 04 Nov 2014 which looks quite > > similar to this problem, but for AMQ 5.9. But there was no answer to his > > question. > > > > -- > > "If you dig a hole and it’s in the wrong place digging it deeper is not > > going to help." – Seymour Chwast > > > > Michiel Hendriks, MSc > > MP Objects - Supply Chain Systems > > > > ✉️ michiel.hendr...@mp-objects.com <javascript:;> > > 🏠 www.mp-objects.com > > ☎ +31 102900304 > > > -- "If you dig a hole and it’s in the wrong place digging it deeper is not going to help." – Seymour Chwast Michiel Hendriks, MSc MP Objects - Supply Chain Systems ✉️ michiel.hendr...@mp-objects.com 🏠 www.mp-objects.com ☎ +31 102900304
