For me, the weirdest part about this is the fact that a full GC somehow fixes is. GCs aren't supposed to do anything that would affect the application (garbage should always be dead, so there should be no effect to anything that's still alive), so that's really strange.
If you start ActiveMQ with a debug port enabled, you can always grab the 5.4.2 source code and attach a debugger once you get into this situation and see if stepping through the code gives any clues to how you got there (and whether the code that "should have" returned values from method calls really is)... And I hope you're planning that upgrade for as soon as you can arrange it. Tim On Wednesday, August 19, 2015, Michiel Hendriks < michiel.hendr...@mp-objects.com> wrote: > We've got a really weird problem. Some times we still have messages on a > queue, but they are not being consumed by the active consumers. > > Via JMX the QueueSize is 1 > doesCursorHaveMessagesBuffered return false > cursorSize returns 0 > doesCursorHaveSpace returns true > I can browse the message > > Restarting consumers has no effect. Restarting ActiveMQ, or simply > performing a gc on the broker via JMX will correct it. Sending a message to > the queue will push the current pending message to a consumer, but the new > message is then stuck. > > I've created and analyzed a memory dump. The batchList (OrderedPendingList) > of the QueueStorePrefetch contains a map with size 1, so > doesCursorHaveMessagesBuffered should have returned true rather than false. > > We have been unable to create a reproduction case yet, it just happens at > some point at stays like that until a restart or explicit gc on the broker. > The queue has 20 consumers (prefetch=1), and a lot of messages do need to > be redelivered. > > This is with ActiveMQ 5.4.2 running on java 1.6.0_36 > (rhel-1.13.8.1.el5_11-x86_64). Yes, I know it's rather old (we don't have > the luxury yet to upgrade). Looking at the code of a newer ActiveMQ I do > not see differences in the path from doesCursorHaveMessagesBuffered to > OrderedPendingList.isEmpty > > Has anybody seen anything like this before? > > I did see a message from Donnell Alwyn on 04 Nov 2014 which looks quite > similar to this problem, but for AMQ 5.9. But there was no answer to his > question. > > -- > "If you dig a hole and it’s in the wrong place digging it deeper is not > going to help." – Seymour Chwast > > Michiel Hendriks, MSc > MP Objects - Supply Chain Systems > > ✉️ michiel.hendr...@mp-objects.com <javascript:;> > 🏠 www.mp-objects.com > ☎ +31 102900304 >
