After I say you wrote 'prefetchExtension=false' I looked it up and found this bug sounds exactly like what I'm hitting: https://issues.apache.org/jira/browse/AMQ-2651 which led me to you talking on http://grokbase.com/t/activemq/users/103bdh5cgx/prefetchextension-off-by-1-for-transacted-consumers-with-prefetchsize-0
So... right now I have prefetch=1.... and I'm using 5.3.0. WIth the grails jms (spring) plugin, its auto-ack for messages, and they are in a transaction. So it sounds like I'm hitting this. Does prefetchExtension=false exist in 5.3? (Looks like it was fixed in 5.4) Should I really be using prefetch=0? In this one queue, I have 16 listeners now, and messages are usually in groups < 10 but take a long time to process. (hours) On Wed, Nov 13, 2013 at 4:27 PM, Gary Tully <gary.tu...@gmail.com> wrote: > can you try a different ack mode, like clientack or using transactions > - the prefetch will be deferred till the ack which will be later than > in the auto ack case. Also, in the transacted case, use the > destination policy prefetchExtension=false > > On 13 November 2013 14:54, Ned Wolpert <ned.wolp...@imemories.com> wrote: > > Did anyone have an idea into what I could do different to route messages > to > > idle consumers? Just came into the same situation this morning where a > > queue has 1 message processing on one consumer, one message waiting, and > 15 > > idle consumers. (See notes below for my current configs) > > > > > > On Wed, Nov 6, 2013 at 9:40 AM, Ned Wolpert <ned.wolp...@imemories.com > >wrote: > > > >> Forgot to add, broker url only has one query param.... > >> > >> jms.prefetchPolicy.queuePrefetch=1 > >> > >> which, as I mentioned above, does seem to work. > >> > >> > >> On Tue, Nov 5, 2013 at 10:56 AM, Ned Wolpert <ned.wolp...@imemories.com > >wrote: > >> > >>> I can see the preFetch values being set in the console, and they are > all > >>> one. I've not set priorities. > >>> > >>> These are 'java' processes, using groovy/grails. The same executable > on 4 > >>> boxes, each executable with 4 listeners, treaded. Using the grails jms > >>> plugin, which wraps the Spring jms template configuration. > >>> (concurrentConsumers is set to 4 per instance) > >>> > >>> When I have 1000's of messages pending, all instances are working. This > >>> issue is only really viewable when there is 10 messages working. > >>> > >>> The following is the (redacted) activemq.xml. I'm assuming this config > >>> could be better. I should mention typical usage of our JMS server has > a > >>> few consumers and tons of producers. Thirty queues. Most queues process > >>> quickly and do not fill up. Two queues are for slow producers. The > goal is > >>> for the producers to send a message and break away, so we don't want > slow > >>> producers at all. Producers are very spiky.... from 10m/min to bursts > of > >>> 100's/min. We have growth concern as that number is increasing > steadily. > >>> > >>> <beans > >>> xmlns="http://www.springframework.org/schema/beans"; > >>> xmlns:amq="http://activemq.apache.org/schema/core"; > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > >>> xsi:schemaLocation="http://www.springframework.org/schema/beans > >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > >>> http://activemq.apache.org/schema/core > >>> http://activemq.apache.org/schema/core/activemq-core.xsd";> > >>> > >>> <bean > >>> > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > >>> <property name="locations"> > >>> <value>file:${activemq.base}/conf/credentials.properties</value> > >>> </property> > >>> </bean> > >>> > >>> <broker xmlns="http://activemq.apache.org/schema/core"; > >>> brokerName="stagingMQ" > >>> useJmx="true" > >>> enableStatistics="true" > >>> useLocalHostBrokerName="false" > >>> useLoggingForShutdownErrors="true" > >>> dataDirectory="XXXXX"> > >>> > >>> <managementContext> > >>> <managementContext createConnector="true" > >>> connectorPort="XXXXX"/> > >>> </managementContext> > >>> > >>> <persistenceAdapter> > >>> <journaledJDBC journalLogFiles="5" > >>> journalLogFileSize="20 Mb" > >>> dataDirectory="XXXXXX" > >>> createTablesOnStartup="false" > >>> useDatabaseLock="false" > >>> dataSource="#XXXXX"> > >>> </journaledJDBC> > >>> </persistenceAdapter> > >>> > >>> <destinationPolicy> > >>> <policyMap> > >>> <policyEntries> > >>> <policyEntry topic=">" producerFlowControl="true" > >>> memoryLimit="1mb"> > >>> <pendingSubscriberPolicy> > >>> <vmCursor /> > >>> </pendingSubscriberPolicy> > >>> </policyEntry> > >>> <policyEntry queue=">" producerFlowControl="true" memoryLimit="30mb"> > >>> <pendingQueuePolicy> > >>> <vmQueueCursor/> > >>> </pendingQueuePolicy> > >>> </policyEntry> > >>> </policyEntries> > >>> </policyMap> > >>> </destinationPolicy> > >>> > >>> <transportConnectors> > >>> <transportConnector name="openwire" uri="XXXX"/> > >>> <transportConnector name="stomp" uri="XXXXX"/> > >>> </transportConnectors> > >>> > >>> </broker> > >>> > >>> <import resource="jetty.xml"/> > >>> <import resource="databaseconfig.xml"/> > >>> </beans> > >>> > >>> > >>> > >>> > >>> On Tue, Nov 5, 2013 at 9:27 AM, Paul Gale <paul.n.g...@gmail.com> > wrote: > >>> > >>>> Have you verified via broker logging that the prefetch values you've > >>>> configured are being honored by the broker? Are consumer priorities in > >>>> use? Are your consumers instances of the same executable or are they > >>>> implemented individually? > >>>> > >>>> Can you post your broker configuration: activemq.xml? > >>>> > >>>> How are your clients implemented, e.g., technology: Ruby or Java etc, > >>>> choice of client libraries? Just wondering. > >>>> > >>>> > >>>> Thanks, > >>>> Paul > >>>> > >>>> On Tue, Nov 5, 2013 at 10:28 AM, Ned Wolpert < > ned.wolp...@imemories.com> > >>>> wrote: > >>>> > Thanks for the response... > >>>> > > >>>> > Any idea on the round-robin not working? I have a queue with 16 > >>>> consumers, > >>>> > all have pre-fetch set to 1. Five consumers are actively processing > >>>> > requests and 3 requests are pending.... the 11 other consumers are > >>>> idle. > >>>> > History has shown that a new request may go to one of the 11 idle > >>>> works, > >>>> > but its like those 3 requests are reserved for some of the working > >>>> ones. I > >>>> > can't figure out what setting would help this, or if this just was a > >>>> bug > >>>> > with 5.3.... > >>>> > > >>>> > > >>>> > On Mon, Nov 4, 2013 at 4:37 PM, Christian Posta > >>>> > <christian.po...@gmail.com>wrote: > >>>> > > >>>> >> The clients should negotiate the correct open-wire (protocol > version) > >>>> >> so in theory the broker will be backward compatible with older > >>>> >> clients. Just make sure the activemq-openwire-legacy jar is on the > >>>> >> classpath (should be by default). > >>>> >> > >>>> >> Of course I would test this out to make sure :) > >>>> >> > >>>> >> On Mon, Nov 4, 2013 at 10:20 AM, Ned Wolpert < > >>>> ned.wolp...@imemories.com> > >>>> >> wrote: > >>>> >> > Folks- > >>>> >> > > >>>> >> > I have a 5.3 installation that we're using, and I have 2 > >>>> questions for > >>>> >> it: > >>>> >> > > >>>> >> > 1) We have prefetch set to 1 for all of the message consumers on > one > >>>> >> queue, > >>>> >> > where message handling is slow. But it still seems like messages > >>>> aren't > >>>> >> > really 'round robin' to the next available message consumer. I'll > >>>> see a > >>>> >> few > >>>> >> > consumers are free but messages are waiting around. Is there a > >>>> >> > configuration that can help? (I should note that the server has > >>>> been > >>>> >> > running consistently for 9 months and it seems to be getting > >>>> worse.... > >>>> >> > would a restart help?) > >>>> >> > > >>>> >> > 2) We are looking to upgrade to 5.9. I haven't started the > process > >>>> of > >>>> >> > testing, but I wanted to see if this is a case where the 5.3 > >>>> clients need > >>>> >> > to be upgraded at the same time as the server, or if the clients > >>>> can be > >>>> >> > rolled over a few weeks to 5.9 after the server gets updated? > >>>> >> > > >>>> >> > Thanks! > >>>> >> > > >>>> >> > -- > >>>> >> > Virtually, Ned Wolpert > >>>> >> > > >>>> >> > "Settle thy studies, Faustus, and begin..." --Marlowe > >>>> >> > >>>> >> > >>>> >> > >>>> >> -- > >>>> >> Christian Posta > >>>> >> http://www.christianposta.com/blog > >>>> >> twitter: @christianposta > >>>> >> > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > Virtually, Ned Wolpert > >>>> > > >>>> > "Settle thy studies, Faustus, and begin..." --Marlowe > >>>> > >>> > >>> > >>> > >>> -- > >>> Virtually, Ned Wolpert > >>> > >>> "Settle thy studies, Faustus, and begin..." --Marlowe > >>> > >> > >> > >> > >> -- > >> Virtually, Ned Wolpert > >> > >> "Settle thy studies, Faustus, and begin..." --Marlowe > >> > > > > > > > > -- > > Virtually, Ned Wolpert > > > > "Settle thy studies, Faustus, and begin..." --Marlowe > > > > -- > http://redhat.com > http://blog.garytully.com > -- Virtually, Ned Wolpert "Settle thy studies, Faustus, and begin..." --Marlowe
