Hi Rob, Thanks for the reply. I'll give that article a read and see if I can post some client code.
Geir: we're in the process of testing the same setup with Fuse Message Broker, which should give us some indication of where the problem lies. Regards, Maarten rajdavies wrote: > > I think we'd really need to see the client code for the consumer too - > its likely that the consumers are not acknowledging messages properly > - so no more messages will be dispatched - as the broker thinks that > all the consumers are still busy processing. > > Things to be aware of when using Spring's JmsTemplate - > http://activemq.apache.org/jmstemplate-gotchas.html > and here http://activemq.apache.org/spring-support.html - but this > article is extremely informative too - > http://codedependents.com/2009/10/16/efficient-lightweight-jms-with-spring-and-activemq/ > > cheers, > > Rob > On 18 Jan 2010, at 09:44, Maarten_D wrote: > >> >> Anyone have any ideas? >> >> >> Maarten_D wrote: >>> >>> Hi, >>> I've somehow gotten myself into the situation where I've got 50,000+ >>> messages stuck in a queue for no apparent reason. Allow me to >>> explain: >>> >>> I've got a system where a component rapidly produces messages that >>> are put >>> on topic (around 400 per second). A second component listens to the >>> topic, >>> takes the information from the messages that it sees, repackages the >>> information in another message which it puts a on queue. A third >>> component >>> eats from the queue, and processes the information in the messages. >>> >>> Under large load (upwards of 1.5m messages) we were experiencing >>> system >>> hangs (no more messages were getting through), so I let loose the >>> usual >>> stable of performance analysis tools (JConsole and Sar) to see what >>> was >>> going one. Using the graphs produced with the sar info, you can >>> clearly >>> see the points at which the producer flow control kicks in for the >>> topic: >>> the cpu io-wait skyrockets and the JMX counters for the topic grind >>> to a >>> halt. What's troubling, however, is that nothing else seems to be >>> working >>> either (ie the downstream queues that were filled from the topic >>> don't >>> seem to be emptied). >>> >>> Things got even stranger when I killed the producer, thereby >>> cutting the >>> influx of new messages to the topic. The system is now in a stable >>> state, >>> with the amount of enqueued messages on the topic equal to the >>> dequeued >>> amount. However, there are more than 50,000 messages in the first >>> queue, >>> waiting to be processed. All of the listeners that are supposed to >>> eat >>> from this queue are blocked with the following stacktrace: >>> >>> java.lang.Thread.State: TIMED_WAITING on java.lang.obj...@6e186c3f >>> at java.lang.Object.wait(Native Method) >>> at >>> org >>> .apache >>> .activemq >>> .MessageDispatchChannel.dequeue(MessageDispatchChannel.java:77) >>> at >>> org >>> .apache >>> .activemq >>> .ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:428) >>> at >>> org >>> .apache >>> .activemq >>> .ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:554) >>> at >>> org >>> .springframework >>> .jms >>> .listener >>> .AbstractPollingMessageListenerContainer >>> .receiveMessage(AbstractPollingMessageListenerContainer.java:405) >>> at >>> org >>> .springframework >>> .jms >>> .listener >>> .AbstractPollingMessageListenerContainer >>> .doReceiveAndExecute(AbstractPollingMessageListenerContainer.java: >>> 308) >>> at >>> org >>> .springframework >>> .jms >>> .listener >>> .AbstractPollingMessageListenerContainer >>> .receiveAndExecute(AbstractPollingMessageListenerContainer.java:261) >>> at >>> org.springframework.jms.listener.DefaultMessageListenerContainer >>> $ >>> AsyncMessageListenerInvoker >>> .invokeListener(DefaultMessageListenerContainer.java:982) >>> at >>> org.springframework.jms.listener.DefaultMessageListenerContainer >>> $ >>> AsyncMessageListenerInvoker >>> .executeOngoingLoop(DefaultMessageListenerContainer.java:974) >>> at >>> org.springframework.jms.listener.DefaultMessageListenerContainer >>> $ >>> AsyncMessageListenerInvoker >>> .run(DefaultMessageListenerContainer.java:876) >>> at java.lang.Thread.run(Thread.java:619) >>> >>> I've used JConsole to stop and start the (tcp) connector several >>> times, >>> and each time (strangely) around 2075 messages have been eaten from >>> the >>> queue by the consumers, after which things freeze again. If I >>> restart the >>> entire broker, around 800 messages are eaten from the queue before >>> things >>> stagnate again. >>> >>> My basic question is: what is going on, and how can I prevent those >>> messages from getting stuck in the queue? >>> >>> To make matters even more interesting, I ran another test a while ago >>> where 10 million messages were fed to the same setup without a >>> hitch. The >>> only difference between that test and the one I'm running now was the >>> nature of the message. The average size of a message in both sets is >>> almost equal, but during the current test message sizes vary more >>> than >>> during the succesfull one. >>> >>> Anyone have any ideas? Below are a bunch of relevant settings an my >>> activemq.xml config. We code to activemq using the Spring >>> JMSTemplate. >>> >>> Thanks in advance, >>> Maarten >>> >>> ActiveMQ: 5.3.0 >>> Java: 1.6.0_17 >>> Spring: 2.5.6 >>> Connector URL: tcp://localhost:61616 >>> JMS receivetimeout: 30000 >>> JMS Acknowledgemode: CLIENT_ACKNOWLEDGE >>> JMS Session transacted: false >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"; >>> xmlns:amq="http://activemq.apache.org/schema/core"; >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>> xsi:schemaLocation="http://www.springframework.org/schema/beans >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >>> http://activemq.apache.org/schema/core >>> http://activemq.apache.org/schema/core/activemq-core.xsd >>> http://mortbay.com/schemas/jetty/1.0 http://jetty.mortbay.org/jetty.xsd >>> "> >>> >>> <broker id="broker" useJmx="true" brokerName="testbroker" >>> start="true" >>> xmlns="http://activemq.apache.org/schema/core"; >>> dataDirectory="/var/amqdata"> >>> >>> <destinationPolicy> >>> <policyMap> >>> <policyEntries> >>> <policyEntry queue=">" memoryLimit="32 mb" >>> producerFlowControl="true" /> >>> <policyEntry topic=">" memoryLimit="32 mb" >>> producerFlowControl="true" /> >>> </policyEntries> >>> </policyMap> >>> </destinationPolicy> >>> >>> <managementContext> >>> <managementContext >>> useMBeanServer="true" >>> jmxDomainName="org.apache.activemq" >>> createMBeanServer="true" >>> createConnector="false" >>> connectorPort="1100" >>> connectorPath="/jmxrmi"/> >>> </managementContext> >>> >>> <persistenceAdapter> >>> <amqPersistenceAdapter >>> syncOnWrite="false" >>> directory="/var/amqdata/testbroker" >>> indexBinSize="8192" >>> cleanupInterval="300000" >>> indexPageSize="64 kb" >>> maxFileLength="256 mb" >>> archiveDataLogs="false"/> >>> </persistenceAdapter> >>> >>> <systemUsage> >>> <systemUsage> >>> <memoryUsage> >>> <memoryUsage limit="512 mb" /> >>> </memoryUsage> >>> </systemUsage> >>> </systemUsage> >>> >>> <transportConnectors> >>> <transportConnector uri="tcp://localhost:61616" /> >>> </transportConnectors> >>> >>> </broker> >>> >>> </beans> >>> >> >> -- >> View this message in context: >> http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27208079.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > Rob Davies > http://twitter.com/rajdavies > I work here: http://fusesource.com > My Blog: http://rajdavies.blogspot.com/ > I'm writing this: http://www.manning.com/snyder/ > > > > > > > -- View this message in context: http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27210211.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
