This sounds very much like the problem I was having a while ago - I was doing some stress testing, and could get ActiveMQ to lock up fairly quickly. IIRC, I was getting messages regarding the much-dreaded producer flow control.
On a lark, I swapped and used OpenMQ to see what would happen and it just worked as expected, making me fairly sure it wasn't my producer or client code. Can you give that a shot and see what happens? geir On Jan 18, 2010, at 4:44 AM, Maarten_D wrote: > > Anyone have any ideas? > > > Maarten_D wrote: >> >> Hi, >> I've somehow gotten myself into the situation where I've got 50,000+ >> messages stuck in a queue for no apparent reason. Allow me to explain: >> >> I've got a system where a component rapidly produces messages that are put >> on topic (around 400 per second). A second component listens to the topic, >> takes the information from the messages that it sees, repackages the >> information in another message which it puts a on queue. A third component >> eats from the queue, and processes the information in the messages. >> >> Under large load (upwards of 1.5m messages) we were experiencing system >> hangs (no more messages were getting through), so I let loose the usual >> stable of performance analysis tools (JConsole and Sar) to see what was >> going one. Using the graphs produced with the sar info, you can clearly >> see the points at which the producer flow control kicks in for the topic: >> the cpu io-wait skyrockets and the JMX counters for the topic grind to a >> halt. What's troubling, however, is that nothing else seems to be working >> either (ie the downstream queues that were filled from the topic don't >> seem to be emptied). >> >> Things got even stranger when I killed the producer, thereby cutting the >> influx of new messages to the topic. The system is now in a stable state, >> with the amount of enqueued messages on the topic equal to the dequeued >> amount. However, there are more than 50,000 messages in the first queue, >> waiting to be processed. All of the listeners that are supposed to eat >> from this queue are blocked with the following stacktrace: >> >> java.lang.Thread.State: TIMED_WAITING on java.lang.obj...@6e186c3f >> at java.lang.Object.wait(Native Method) >> at >> org.apache.activemq.MessageDispatchChannel.dequeue(MessageDispatchChannel.java:77) >> at >> org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:428) >> at >> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:554) >> at >> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:405) >> at >> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308) >> at >> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261) >> at >> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982) >> at >> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974) >> at >> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876) >> at java.lang.Thread.run(Thread.java:619) >> >> I've used JConsole to stop and start the (tcp) connector several times, >> and each time (strangely) around 2075 messages have been eaten from the >> queue by the consumers, after which things freeze again. If I restart the >> entire broker, around 800 messages are eaten from the queue before things >> stagnate again. >> >> My basic question is: what is going on, and how can I prevent those >> messages from getting stuck in the queue? >> >> To make matters even more interesting, I ran another test a while ago >> where 10 million messages were fed to the same setup without a hitch. The >> only difference between that test and the one I'm running now was the >> nature of the message. The average size of a message in both sets is >> almost equal, but during the current test message sizes vary more than >> during the succesfull one. >> >> Anyone have any ideas? Below are a bunch of relevant settings an my >> activemq.xml config. We code to activemq using the Spring JMSTemplate. >> >> Thanks in advance, >> Maarten >> >> ActiveMQ: 5.3.0 >> Java: 1.6.0_17 >> Spring: 2.5.6 >> Connector URL: tcp://localhost:61616 >> JMS receivetimeout: 30000 >> JMS Acknowledgemode: CLIENT_ACKNOWLEDGE >> JMS Session transacted: false >> >> <beans xmlns="http://www.springframework.org/schema/beans"; >> xmlns:amq="http://activemq.apache.org/schema/core"; >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >> http://activemq.apache.org/schema/core >> http://activemq.apache.org/schema/core/activemq-core.xsd >> http://mortbay.com/schemas/jetty/1.0 http://jetty.mortbay.org/jetty.xsd";> >> >> <broker id="broker" useJmx="true" brokerName="testbroker" start="true" >> xmlns="http://activemq.apache.org/schema/core"; >> dataDirectory="/var/amqdata"> >> >> <destinationPolicy> >> <policyMap> >> <policyEntries> >> <policyEntry queue=">" memoryLimit="32 mb" >> producerFlowControl="true" /> >> <policyEntry topic=">" memoryLimit="32 mb" >> producerFlowControl="true" /> >> </policyEntries> >> </policyMap> >> </destinationPolicy> >> >> <managementContext> >> <managementContext >> useMBeanServer="true" >> jmxDomainName="org.apache.activemq" >> createMBeanServer="true" >> createConnector="false" >> connectorPort="1100" >> connectorPath="/jmxrmi"/> >> </managementContext> >> >> <persistenceAdapter> >> <amqPersistenceAdapter >> syncOnWrite="false" >> directory="/var/amqdata/testbroker" >> indexBinSize="8192" >> cleanupInterval="300000" >> indexPageSize="64 kb" >> maxFileLength="256 mb" >> archiveDataLogs="false"/> >> </persistenceAdapter> >> >> <systemUsage> >> <systemUsage> >> <memoryUsage> >> <memoryUsage limit="512 mb" /> >> </memoryUsage> >> </systemUsage> >> </systemUsage> >> >> <transportConnectors> >> <transportConnector uri="tcp://localhost:61616" /> >> </transportConnectors> >> >> </broker> >> >> </beans> >> > > -- > View this message in context: > http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27208079.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
