Hi Rob, Sorry, I'd killed that particular JVM before I read your request, and the error hasn't reoccurred since then. Something else that I've been thinking about since reading a http://tmielke.blogspot.com/2009/12/using-spring-jms-template-for-sending.html blog post : we used the ActiveMQ PooledConnectionFactory for establishing all connections to the broker, for the Spring message listener containers as well as for the JMS templates. After diving into the code of the PooledconnectionFactory, this seems to have been a bad idea. When a connection is requested, the PCF simply creates a new one and hands it out. When the pool is full, it returns the first connection in its list (ie the first one it created), removes it from the top of the list and adds it to the bottom. This means that, if your listeners also get connections from this pool (and remember, listeners hold on to their connections), a whole bunch of JMS template calls will be made where the template is sending messages using the same connector used by a listener. I'm not too sure about the details, but when you introduce producerflowcontrol into this picture, I can imagine how a kind of deadlock can occur where eventually all producers on all connections are throttled, leaving no one able to send any messages. Does this sound like a plausible scenario?
We've modified our config and given all listener containers a connection that's not in the pool, and are now running another test. I'll post the results. Regards, Maarten PS. Of course, you might say we should've know about this beforehand, as the PCF javadoc says that its not really meant for consumers. The alternative it offers is Jencks, although that project has been dead for a while, to the point where even the homepage is now a spam site (jencks.org). So clearly that isn't a viable alternative. rajdavies wrote: > > Can you take a thread dump whilst its in this state - and send us the > output ? > > thanks, > > Rob > On 21 Jan 2010, at 17:26, Maarten_D wrote: > >> >> O, and I forgot to mention I also turned on async sends >> (jms.useAsyncSend=true) >> >> Maarten_D wrote: >>> >>> I've now changed my activemq.xml to the listing below, made the >>> session >>> transacted and set the acknowledge mode to SESSION_TRANSACTED. >>> >>> Things were going well for me for a while, with the system >>> processing 3,2 >>> million messages without a hitch, and then everything stopped >>> because the >>> first component in the chain got lots of these: >>> >>> javax.jms.InvalidClientIDException: Broker: broker - Client: >>> ID:rhost-59116-1263927611185-1:445 already connected from / >>> 127.0.0.1:56560 >>> >>> And for an hour now, since it stopped processing messages, the >>> broker has >>> been eating up almost 100% of the cpu for some reason I can't quite >>> fathom >>> (disk utilization is very low, and there is no message traffic >>> passing >>> through the broker). >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"; >>> xmlns:amq="http://activemq.apache.org/schema/core"; >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>> xsi:schemaLocation="http://www.springframework.org/schema/beans >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >>> http://activemq.apache.org/schema/core >>> http://activemq.apache.org/schema/core/activemq-core.xsd >>> http://mortbay.com/schemas/jetty/1.0 http://jetty.mortbay.org/jetty.xsd >>> "> >>> >>> <bean >>> class >>> = >>> "org >>> .springframework.beans.factory.config.PropertyPlaceholderConfigurer"> >>> <property name="location" value="file:/etc/broker.properties" /> >>> </bean> >>> >>> <broker id="broker" useJmx="true" brokerName="${broker.name}" >>> start="true" xmlns="http://activemq.apache.org/schema/core"; >>> dataDirectory="${activemq.data}"> >>> >>> <destinationPolicy> >>> <policyMap> >>> <policyEntries> >>> <policyEntry queue=">" memoryLimit="32mb" >>> strictOrderDispatch="true" producerFlowControl="false"> >>> <pendingQueuePolicy> >>> <vmQueueCursor /> >>> </pendingQueuePolicy> >>> </policyEntry> >>> <policyEntry topic=">" memoryLimit="32mb" >>> producerFlowControl="true" /> >>> </policyEntries> >>> </policyMap> >>> </destinationPolicy> >>> >>> <managementContext> >>> <managementContext useMBeanServer="true" >>> jmxDomainName="org.apache.activemq" >>> createMBeanServer="true" >>> createConnector="false" >>> connectorPort="1100" >>> connectorPath="/jmxrmi"/> >>> </managementContext> >>> >>> <persistenceAdapter> >>> <kahaDB directory="${activemq.data}/${broker.name}" >>> journalMaxFileLength="32mb" >>> enableJournalDiskSyncs="false" >>> indexWriteBatchSize="1000" >>> indexCacheSize="1000"/> >>> </persistenceAdapter> >>> >>> <systemUsage> >>> <systemUsage> >>> <memoryUsage> >>> <memoryUsage limit="512mb" /> >>> </memoryUsage> >>> </systemUsage> >>> </systemUsage> >>> >>> <transportConnectors> >>> <transportConnector uri="nio://0.0.0.0:61616" /> >>> </transportConnectors> >>> </broker> >>> >>> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0";> >>> <connectors> >>> <nioConnector port="61617"/> >>> </connectors> >>> <handlers> >>> <webAppContext contextPath="/admin" >>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/> >>> </handlers> >>> </jetty> >>> </beans> >>> >> >> -- >> View this message in context: >> http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27261393.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > Rob Davies > http://twitter.com/rajdavies > I work here: http://fusesource.com > My Blog: http://rajdavies.blogspot.com/ > I'm writing this: http://www.manning.com/snyder/ > > > > > > > -- View this message in context: http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27592645.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
