We use ActiveMQ 5.14.1 with KahaDB for persistence. >From last 2 days, we have observed that the broker becomes unresponsive after running for around 4 hours! None of the operations work after this including establishing new connections, publishing messages to topics, subscribing to topics, unsubscribing, etc.
We have disabled JMX for performance reasons; so I cannot check the status/health of broker from JMX. I tried to take thread dump to see what's happening, but it fails with a message : Unable to open socket file: target process not responding or HotSpot VM not loaded! Similar error when I try to take heap dump! But I can see that broker process is running using ps -ef |grep java option. Tried to take the thread dump forcefully using the -F option, but this also fails with "java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f9288012800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)" Forceful Heap dump fails with "Expecting GenCollectedHeap, G1CollectedHeap, or ParallelScavengeHeap, but got sun.jvm.hotspot.gc_interface.CollectedHeap" We have just one broker running on AWS EC2 Ubuntu instance. Broker is started with Xmx of 12 GB. Our server and Android applications together create thousands of topics and exchange MQTT messages (both persistent and non-persistent). Within 4 hours, around 20 GB of journal files got created in the last run before broker became unresponsive! The only way to overcome this problem is to stop the broker, delete all files in KahaDB and restart the broker! Any hints to what could be going wrong is highly appreciated! Broker configurations is given below for reference : <broker xmlns="http://activemq.apache.org/schema/core"; useJmx="false" brokerName="PrimaryBroker" deleteAllMessagesOnStartup="false" advisorySupport="false" schedulePeriodForDestinationPurge="600000" offlineDurableSubscriberTimeout="54000000" offlineDurableSubscriberTaskSchedule="3600000" dataDirectory="${activemq.data}"> <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=">" gcInactiveDestinations="true" inactiveTimoutBeforeGC="3600000"> <pendingMessageLimitStrategy> <constantPendingMessageLimitStrategy limit="1000"/> </pendingMessageLimitStrategy> <deadLetterStrategy> <sharedDeadLetterStrategy processExpired="false" /> </deadLetterStrategy> </policyEntry> <policyEntry queue=">" optimizedDispatch="true" reduceMemoryFootprint="true"> <deadLetterStrategy> <sharedDeadLetterStrategy processExpired="false" /> </deadLetterStrategy> </policyEntry> </policyEntries> </policyMap> </destinationPolicy> <persistenceAdapter> <kahaDB directory="${activemq.data}/kahadb" indexCacheSize="20000" enableJournalDiskSyncs="false" ignoreMissingJournalfiles="true"/> </persistenceAdapter> <systemUsage> <systemUsage sendFailIfNoSpaceAfterTimeout="10000"> <memoryUsage> <memoryUsage percentOfJvmHeap="70" /> </memoryUsage> <storeUsage> <storeUsage limit="100 gb"/> </storeUsage> <tempUsage> <tempUsage limit="50 gb"/> </tempUsage> </systemUsage> </systemUsage> <transportConnectors> <transportConnector name="openwire" uri="nio://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxInactivityDuration=180000&wireFormat.maxFrameSize=104857600"/> <transportConnector name="mqtt+nio" uri="mqtt+nio://0.0.0.0:1883?maximumConnections=50000&wireFormat.maxInactivityDuration=180000&wireFormat.maxFrameSize=104857600"/> </transportConnectors> <plugins> <discardingDLQBrokerPlugin dropAll="true" dropTemporaryTopics="true" dropTemporaryQueues="true" /> <timeStampingBrokerPlugin ttlCeiling="43200000" zeroExpirationOverride="43200000"/> </plugins> </broker> TIA, Shobhana -- View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-broker-becomes-unresponsive-after-sometime-tp4725278.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
