Hi Rob, I'll describe what I see from a heap dump during a run of the test Dan described. I'm running the 0.32 broker and client, though I've also looked at a heap dump with the 0.16 broker and client and have seen similar patterns.
The "steady state" heap size of my broker is ~200MB. After running the test, the heap goes up to ~1GB, even though there are no more messages on queue A and I have manually triggered a GC. What I see is that the PriorityQueueImpl that represents Queue B from the test has a retained size that takes up most of the 1GB. It appears that from the steps 3(b)-(e) that Dan described, objects representing the consumers of both Queue A and Queue B are still being referenced by the entries of Queue B, and these consumer objects of Queue A can reference old entries of the queue that have already been drained in steps 3(f)-(h). More details and an example of this: I attached 2 screenshots of from the heap analysis. qpid_broker_heap_queueB_referenceToQueueAConsumer.png shows the object reference tree of the PriorityQueueImpl representing Queue B. I actually ran the test with Queue B having 20 messages rather than 1, and the screenshot is showing the _stateChangeListeners list of the 4th queue entry. We then see the list holding a QueueConsumerImpl for Queue B, which eventually references its ServerSession, which references its 2 _consumers. The 0 index element of this is for Queue A, and the 1 index element is Queue B. Then if you look at qpid_broker_heap_queueB_queueAEntries.png, you'll see what's referenced by the Queue A consumer. Through _target -> _postIdSettingAction -> _action_entry, we have a reference to a PriorityQueueEntry. This further more starts a linked list of many more entries on Queue A. The _state fields of these entries are all MessageInstance$DeletedState, since we've already drained Queue A. So it seems like in this way, we can be holding onto many queue entries that have already been consumed from the queue. On Wed, Oct 29, 2014 at 3:32 PM, xiaodan.wang <[email protected]> wrote: > Hi Rob, apologies for taking so long to follow-up. We now have meaningful > data (both a reproducible test and heap dump) to describe the issue. I will > first describe the setup of the test we used to reproduce this issue and > then Helen, on of my teammate, will describe the heap dump analysis. > > To recap, we observed that the Qpid (Java) broker heap gradually trended up > in production when we had a handful of messages on a special "suspend" > queue > that were unconsumed. Even though we found no other messages when we > inspected the broker, attempts to GC and reclaim broke memory failed. That > is until we manually deleted the handful of suspended/unconsumed messages. > > I will now describe the test we used to reproduce this issue. We are using > non-persistent messages, asynchronous onMessage delivery, and transacted > sessions. We also ran the test using both the v0.16 Java broker and the > v0.32/trunk Java broker and verified that the issue occurs on both. > > Experiment: > 1) We setup two queues, A and B. > > 2) We first enqueue one message to queue B and do not consume the message > (this message remains unconsumed for the entire duration of the test) > > 3) Next we setup a single test thread that runs iteratively and does: > a) First enqueues 1000 messages to queue A and commits > b) Next, checks out a NEW qpid session and creates a consumer for queue > A. Invokes consumer.receive to fetch one message and then immediately rolls > back the receive. > c) Close the consumer for queue A > d) Using the SAME qpid session from step b), create a consumer for queue > B. Invoke consumer.receive and to fetch one message and then immediately > rolls back the receive. > e) Close the consumer for queue B and close the qpid session. > f) Create a NEW qpid session, and create a consumer for queue A. > g) Invoke consumer.receive repeatedly to drain all 1000 messages from > queue A and then commit. > h) Close the qpid session, at this point there are no messages on queue > A > and a single message on queue B. > > 4) Repeat step 3) many times > > 5) Before test is finished, validate that queue B has a single message and > queue A is empty. > > At the conclusion of the test, we observed that Qpid broker heap is nearly > exhausted and memory cannot be reclaimed via GC. Also, we can reproduce the > same issue if steps 3b) - 3e) is replaced with session.createBrowser (queue > browse/peeking) instead of consumer.receive on queues A and B. > > Helen is in the process of writing up the heap dump analysis and will post > shortly. > Thanks! Dan > > > > > -- > View this message in context: > http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615932.html > Sent from the Apache Qpid users mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
