Yes. Enqueued - dequeued equals the queue size.
Maarten_D wrote: > > Aha. And enqueued - dequeued corresponds to what's in the queue? > > > Elliot Barlas wrote: >> >> Okay, the issue I am seeing is slightly different then. In my case the >> broker isn't even dispatching some messages, at least according to the >> broker stats. Dequeued = Dispatched and Dispatched < Enqueued. >> >> Thanks, >> Elliot >> >> >> Maarten_D wrote: >>> >>> It appears that our problem had to do with prefetching. We had our >>> prefetch values for queues set fairly high, and when I client >>> application would crash, a bunch of messages that were prefetched, but >>> not ack'ed, would remain stuck in the queue. We added >>> "jms.prefetchPolicy.queuePrefetch=1" to our connection uri, and haven't >>> seen this behaviour since. >>> >>> >>> Elliot Barlas wrote: >>>> >>>> Hey Maarten, I am observing the same behavior in my AMQ application. >>>> Was your problem resolved? I've tried a few different connection >>>> factory approach to no avail :( >>>> >>>> Thanks, >>>> Elliot >>>> >>>> >>>> >>>> Maarten_D wrote: >>>>> >>>>> Hi Rob, >>>>> Sorry, I'd killed that particular JVM before I read your request, and >>>>> the error hasn't reoccurred since then. >>>>> Something else that I've been thinking about since reading a >>>>> http://tmielke.blogspot.com/2009/12/using-spring-jms-template-for-sending.html >>>>> blog post : we used the ActiveMQ PooledConnectionFactory for >>>>> establishing all connections to the broker, for the Spring message >>>>> listener containers as well as for the JMS templates. >>>>> After diving into the code of the PooledconnectionFactory, this seems >>>>> to have been a bad idea. When a connection is requested, the PCF >>>>> simply creates a new one and hands it out. When the pool is full, it >>>>> returns the first connection in its list (ie the first one it >>>>> created), removes it from the top of the list and adds it to the >>>>> bottom. This means that, if your listeners also get connections from >>>>> this pool (and remember, listeners hold on to their connections), a >>>>> whole bunch of JMS template calls will be made where the template is >>>>> sending messages using the same connector used by a listener. >>>>> I'm not too sure about the details, but when you introduce >>>>> producerflowcontrol into this picture, I can imagine how a kind of >>>>> deadlock can occur where eventually all producers on all connections >>>>> are throttled, leaving no one able to send any messages. >>>>> Does this sound like a plausible scenario? >>>>> >>>>> We've modified our config and given all listener containers a >>>>> connection that's not in the pool, and are now running another test. >>>>> I'll post the results. >>>>> >>>>> Regards, >>>>> Maarten >>>>> >>>>> PS. Of course, you might say we should've know about this beforehand, >>>>> as the PCF javadoc says that its not really meant for consumers. The >>>>> alternative it offers is Jencks, although that project has been dead >>>>> for a while, to the point where even the homepage is now a spam site >>>>> (jencks.org). So clearly that isn't a viable alternative. >>>>> >>>>> >>>>> rajdavies wrote: >>>>>> >>>>>> Can you take a thread dump whilst its in this state - and send us the >>>>>> output ? >>>>>> >>>>>> thanks, >>>>>> >>>>>> Rob >>>>>> On 21 Jan 2010, at 17:26, Maarten_D wrote: >>>>>> >>>>>>> >>>>>>> O, and I forgot to mention I also turned on async sends >>>>>>> (jms.useAsyncSend=true) >>>>>>> >>>>>>> Maarten_D wrote: >>>>>>>> >>>>>>>> I've now changed my activemq.xml to the listing below, made the >>>>>>>> session >>>>>>>> transacted and set the acknowledge mode to SESSION_TRANSACTED. >>>>>>>> >>>>>>>> Things were going well for me for a while, with the system >>>>>>>> processing 3,2 >>>>>>>> million messages without a hitch, and then everything stopped >>>>>>>> because the >>>>>>>> first component in the chain got lots of these: >>>>>>>> >>>>>>>> javax.jms.InvalidClientIDException: Broker: broker - Client: >>>>>>>> ID:rhost-59116-1263927611185-1:445 already connected from / >>>>>>>> 127.0.0.1:56560 >>>>>>>> >>>>>>>> And for an hour now, since it stopped processing messages, the >>>>>>>> broker has >>>>>>>> been eating up almost 100% of the cpu for some reason I can't quite >>>>>>>> fathom >>>>>>>> (disk utilization is very low, and there is no message traffic >>>>>>>> passing >>>>>>>> through the broker). >>>>>>>> >>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"; >>>>>>>> xmlns:amq="http://activemq.apache.org/schema/core"; >>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans >>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >>>>>>>> http://activemq.apache.org/schema/core >>>>>>>> http://activemq.apache.org/schema/core/activemq-core.xsd >>>>>>>> http://mortbay.com/schemas/jetty/1.0 >>>>>>>> http://jetty.mortbay.org/jetty.xsd >>>>>>>> "> >>>>>>>> >>>>>>>> <bean >>>>>>>> class >>>>>>>> = >>>>>>>> "org >>>>>>>> .springframework.beans.factory.config.PropertyPlaceholderConfigurer"> >>>>>>>> <property name="location" value="file:/etc/broker.properties" /> >>>>>>>> </bean> >>>>>>>> >>>>>>>> <broker id="broker" useJmx="true" brokerName="${broker.name}" >>>>>>>> start="true" xmlns="http://activemq.apache.org/schema/core"; >>>>>>>> dataDirectory="${activemq.data}"> >>>>>>>> >>>>>>>> <destinationPolicy> >>>>>>>> <policyMap> >>>>>>>> <policyEntries> >>>>>>>> <policyEntry queue=">" memoryLimit="32mb" >>>>>>>> strictOrderDispatch="true" producerFlowControl="false"> >>>>>>>> <pendingQueuePolicy> >>>>>>>> <vmQueueCursor /> >>>>>>>> </pendingQueuePolicy> >>>>>>>> </policyEntry> >>>>>>>> <policyEntry topic=">" memoryLimit="32mb" >>>>>>>> producerFlowControl="true" /> >>>>>>>> </policyEntries> >>>>>>>> </policyMap> >>>>>>>> </destinationPolicy> >>>>>>>> >>>>>>>> <managementContext> >>>>>>>> <managementContext useMBeanServer="true" >>>>>>>> jmxDomainName="org.apache.activemq" >>>>>>>> createMBeanServer="true" >>>>>>>> createConnector="false" >>>>>>>> connectorPort="1100" >>>>>>>> connectorPath="/jmxrmi"/> >>>>>>>> </managementContext> >>>>>>>> >>>>>>>> <persistenceAdapter> >>>>>>>> <kahaDB directory="${activemq.data}/${broker.name}" >>>>>>>> journalMaxFileLength="32mb" >>>>>>>> enableJournalDiskSyncs="false" >>>>>>>> indexWriteBatchSize="1000" >>>>>>>> indexCacheSize="1000"/> >>>>>>>> </persistenceAdapter> >>>>>>>> >>>>>>>> <systemUsage> >>>>>>>> <systemUsage> >>>>>>>> <memoryUsage> >>>>>>>> <memoryUsage limit="512mb" /> >>>>>>>> </memoryUsage> >>>>>>>> </systemUsage> >>>>>>>> </systemUsage> >>>>>>>> >>>>>>>> <transportConnectors> >>>>>>>> <transportConnector uri="nio://0.0.0.0:61616" /> >>>>>>>> </transportConnectors> >>>>>>>> </broker> >>>>>>>> >>>>>>>> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0";> >>>>>>>> <connectors> >>>>>>>> <nioConnector port="61617"/> >>>>>>>> </connectors> >>>>>>>> <handlers> >>>>>>>> <webAppContext contextPath="/admin" >>>>>>>> resourceBase="${activemq.base}/webapps/admin" >>>>>>>> logUrlOnStart="true"/> >>>>>>>> </handlers> >>>>>>>> </jetty> >>>>>>>> </beans> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27261393.html >>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>>>> >>>>>> >>>>>> Rob Davies >>>>>> http://twitter.com/rajdavies >>>>>> I work here: http://fusesource.com >>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > -- View this message in context: http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27714260.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
