Hi Rob, apologies for taking so long to follow-up. We now have meaningful data (both a reproducible test and heap dump) to describe the issue. I will first describe the setup of the test we used to reproduce this issue and then Helen, on of my teammate, will describe the heap dump analysis.
To recap, we observed that the Qpid (Java) broker heap gradually trended up in production when we had a handful of messages on a special "suspend" queue that were unconsumed. Even though we found no other messages when we inspected the broker, attempts to GC and reclaim broke memory failed. That is until we manually deleted the handful of suspended/unconsumed messages. I will now describe the test we used to reproduce this issue. We are using non-persistent messages, asynchronous onMessage delivery, and transacted sessions. We also ran the test using both the v0.16 Java broker and the v0.32/trunk Java broker and verified that the issue occurs on both. Experiment: 1) We setup two queues, A and B. 2) We first enqueue one message to queue B and do not consume the message (this message remains unconsumed for the entire duration of the test) 3) Next we setup a single test thread that runs iteratively and does: a) First enqueues 1000 messages to queue A and commits b) Next, checks out a NEW qpid session and creates a consumer for queue A. Invokes consumer.receive to fetch one message and then immediately rolls back the receive. c) Close the consumer for queue A d) Using the SAME qpid session from step b), create a consumer for queue B. Invoke consumer.receive and to fetch one message and then immediately rolls back the receive. e) Close the consumer for queue B and close the qpid session. f) Create a NEW qpid session, and create a consumer for queue A. g) Invoke consumer.receive repeatedly to drain all 1000 messages from queue A and then commit. h) Close the qpid session, at this point there are no messages on queue A and a single message on queue B. 4) Repeat step 3) many times 5) Before test is finished, validate that queue B has a single message and queue A is empty. At the conclusion of the test, we observed that Qpid broker heap is nearly exhausted and memory cannot be reclaimed via GC. Also, we can reproduce the same issue if steps 3b) - 3e) is replaced with session.createBrowser (queue browse/peeking) instead of consumer.receive on queues A and B. Helen is in the process of writing up the heap dump analysis and will post shortly. Thanks! Dan -- View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615932.html Sent from the Apache Qpid users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
