[
https://issues.apache.org/jira/browse/AMQ-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035411#comment-15035411
]
Jouni Mäki-Panula commented on AMQ-5993:
----------------------------------------
Message priority wasn't used in my case.
Since creating the issue, I've increased system memory to 8 GB and heap size to
2 GB. I'll report if the error recurs, but for now, the system seems to be in
healthier condition, with over 5 GB free memory when counting OS caches and
around 1 GB actual free memory.
> Purging messages leads to GC overhead limit exceeded
> ----------------------------------------------------
>
> Key: AMQ-5993
> URL: https://issues.apache.org/jira/browse/AMQ-5993
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.11.1
> Reporter: Jouni Mäki-Panula
> Assignee: Christopher L. Shannon
> Priority: Minor
>
> Purging a large number of messages from a queue in ActiveMQ 5.11.1 seems to
> lead to GC overhead limit exceeded error state.
> I've witnessed this behavior twice during the last month. In the first case,
> 3 314 226 messages were purged. 12 minutes later, the broker basically halted
> and log started to fill with messages like these:
> {noformat}
> | WARN | Transport Connection to: tcp://172.20.50.174:56005 failed:
> java.io.IOException: GC overhead limit exceeded |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO
> Worker 27070
> | ERROR | Could not accept connection : java.lang.Exception:
> java.lang.OutOfMemoryError: GC overhead limit exceeded |
> org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
> Thread Handler:
> stomp+nio://0.0.0.0:61613?maximumConnections=1400&wireFormat.maxFrameSize=104857600&transport.keepAlive=true&transport.soLinger=30
> | WARN | Async error occurred: |
> org.apache.activemq.broker.TransportConnection.Service | ActiveMQ
> BrokerService[localhost] Task-22496
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> | WARN | Exception occurred processing:
> null: java.lang.OutOfMemoryError: GC overhead limit exceeded |
> org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ
> BrokerService[localhost] Task-22506
> {noformat}
> In the second case, 2 380 871 messages were purged. This time it took a bit
> longer for the garbage collection to kick in, but it did, causing similar
> error state 6 hours and 14 minutes later:
> {noformat}
> | WARN | Async error occurred: |
> org.apache.activemq.broker.TransportConnection.Service | ActiveMQ NIO Worker
> 16937
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:142)[activemq-kahadb-store-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:627)[activemq-kahadb-store-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.store.kahadb.plist.PListStoreImpl.write(PListStoreImpl.java:403)[activemq-kahadb-store-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.store.kahadb.plist.PListImpl.addLast(PListImpl.java:100)[activemq-kahadb-store-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.flushToDisk(FilePendingMessageCursor.java:440)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:231)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1796)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.Queue.orderedCursorAdd(Queue.java:878)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:854)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.Queue.send(Queue.java:733)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.util.LoggingBrokerPlugin.send(LoggingBrokerPlugin.java:275)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:541)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)[activemq-client-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:45)[activemq-client-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:87)[activemq-stomp-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:199)[activemq-stomp-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompSend(ProtocolConverter.java:332)[activemq-stomp-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:245)[activemq-stomp-5.11.1.jar:5.11.1]
> at
> org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:75)[activemq-stomp-5.11.1.jar:5.11.1]
> | WARN | Exception occurred processing:
> null: java.lang.OutOfMemoryError: GC overhead limit exceeded |
> org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ
> BrokerService[localhost] Task-17265
> | WARN | Transport Connection to: tcp://172.20.50.179:50888 failed:
> java.io.IOException: GC overhead limit exceeded |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO
> Worker 17016
> | ERROR | Could not accept connection : java.lang.Exception:
> java.lang.OutOfMemoryError: GC overhead limit exceeded |
> org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
> Thread Handler:
> stomp+nio://0.0.0.0:61613?maximumConnections=1400&wireFormat.maxFrameSize=104857600&transport.keepAlive=true&transport.soLinger=30
> {noformat}
> The only way out seemed to be restarting the broker.
> This ActiveMQ instance is running on a server with 4 GB system memory and 1
> GB heap size for ActiveMQ. Heap wasn't a problem in neither of the cases. In
> normal usage, there is around 2.5 GB free memory when counting OS caches, but
> only around 150 MB actual free memory. OS is RHEL 6.6.
> The most obvious solution may be to simply increase system memory. Another
> workaround would be to dequeue messages from the queue with a consumer
> instead of purging them.
> But still it seems odd that ActiveMQ is able to handle millions of messages
> if they are dequeued, but not if they are purged. Should the purge operation
> be improved somehow? And is increasing system memory the best option here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
