Bad choice of words.. not "destined" but could run out of memory if consumers cannot keep up with producers
On Wed, Dec 12, 2012 at 5:51 PM, Christian Posta <[email protected]>wrote: > So i guess in general i would think that a configuration with no PFC, vm > cursors, and no persistence is destined to run out of memory. > > I wonder if what's happening in your case is these temporary queues are > getting filled with messages that aren't being consumed and then when the > reference hangs around in the Producer exchange, it's making a dent in the > heap space. If this happens enough times it seems like it can OOM. I'll try > to reproduce that on my side. > > > On Wed, Dec 12, 2012 at 7:55 AM, Jerry Cwiklik <[email protected]> wrote: > >> Christian, thanks. The most recent experiment was run with a much smaller >> broker, Xmx=360M. I am trying to reproduce the problem I am seeing in a >> larger system which is not easy to debug. I can't share the large heap >> dump >> as it may contain proprietary data. But thanks for the offer to help >> examine >> it. >> >> I know that the large heap dump looks exactly as the screenshot I attached >> to the initial post. It looks to me that somehow messages in temp queues >> are >> retained in broker memory. I watched this broker OOM and I can say for >> sure >> that none of the live temp queues showed any build up before the OOM. >> >> Assuming for a moment that this problem is induced by an application, what >> could possibly be causing this build up of ProducerBrokerExchange objects >> which hold references to messages in temp queues? Again, I know that there >> is no buildup in live temp queues so this is not the case of a slow >> consumer >> not keeping up with processing msg from a temp queue. >> >> As mentioned in the original post, we run with producerFlowControl=false, >> no >> persistence, and vm cursor. I ran netstat on a machine where the broker is >> running and dont see any FIN_WAIT nor WAIT_CLOSE. There are no messages in >> brokers log saying that memory usage is high. In fact, the jconsole broker >> MBean doesnt show it being short on memory. The memory usage shows as 0! >> Yet, the brokers hits OOM. >> >> -Jerry C >> >> >> >> >> >> >> >> >> -- >> View this message in context: >> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660538.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta > > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
