Update: I think I found problem why I see this behavior. We have bunch wars running on tomcat and sending messages to broker. We have uptime deployment process for all the wars. So when I undeploy war I think activemq connection threads dont shut down properly and thats why I see exceptions like this in broker logs:
2014-02-18 19:54:20,664 | WARN | Transport Connection to: tcp:// 10.190.225.56:39399 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.56:39399@61618 2014-02-18 19:54:20,664 | WARN | Transport Connection to: tcp:// 10.190.225.56:39412 failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.56:39412@61618 2014-02-18 19:54:20,664 | WARN | Transport Connection to: tcp:// 10.190.225.56:39409 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.56:39409@61618 2014-02-18 19:55:42,270 | WARN | Transport Connection to: tcp:// 10.190.133.15:33382 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>1000) long: tcp://10.190.133.15:33382 | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker I have tried tcpProperty: *daemon=true* on service side and on client side. But if I understand right it helps cleaning up threads when JVM shuts down. In my case we dont stop tomcat. All our deployment happens in uptime. Anyother suggestion that I can try? thanks, chirag On Tue, Feb 11, 2014 at 4:46 PM, Chirag Pujara <chiragpuj...@gmail.com>wrote: > I tried again and this time I didnt send bunch of messages at same time. I > noticed that some message gets processed right away but some messges stays > on queue and doesnt get processed. If send few more messges let say 50 > messages and it processes 40 messages and 10 msgs left on queue. I can send > more messages but not all get processed. > > I am attaching config file. I have enabled producerFlowControl and removed > slowConsumerStrategy. I am using activemq-5.10-SNAPSHOT. > > > > On Mon, Feb 10, 2014 at 9:04 PM, Chirag Pujara <chiragpuj...@gmail.com>wrote: > >> i said broker hangs bcs if i tried to send more messages or messages on >> queue doent get processed. i cannot even create queue or send message using >> webconsole. >> >> will try again and rhis time will moniter memory. >> >> On Feb 10, 2014 6:22 PM, "artnaseef" <a...@artnaseef.com> wrote:< >>> >>> Perhaps I misunderstood "Broker Hangs". It helps to have more detailed >>> symptoms. >>> >>> The messages in the DLQ are failing at the consumer - either transactions >>> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE >>> client calling Session.recover(). >>> >>> See here for more details: >>> http://activemq.apache.org/message-redelivery-and-dlq-handling.html >>> >>> You are right about the webconsole -- it will most likely stop >>> functioning >>> once the broker runs out of memory. >>> >>> > >>> > >>> > I have producerFlowControl to false. I seperated my producer and >>> > consumers. >>> > But still I was able to reproduce issue. For some time bunch of >>> messages >>> > started pile up in queue. Then they started to endup in DLQ. In DLQ I >>> saw >>> > most of them have these property: >>> > >>> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery >>> policy >>> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor = >>> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1, >>> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false, >>> > useExponentialBackOff = false, backOffMultiplier = 5.0, >>> redeliveryDelay = >>> > 1000}, cause:null >>> > >>> > Does this thing has anything to do with the issue? >>> > >>> > I was not monitoring heap size when it got stuck. that will be next >>> thing >>> > to do. But I can hit webconsole from that activemq instance. So if it >>> had >>> > memory issues how come console is still working? isn't it same process? >>> > >>> > thanks, >>> > chirag >>> > >>> > >>> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <a...@artnaseef.com> wrote: >>> > >>> >> How about a stack trace on the client? Can you look for the consumer >>> >> threads >>> >> and see what they are doing? >>> >> >>> >> Note that with producers and consumers on the same connection, it's >>> >> possible >>> >> to reach a deadlock if producer-flow-control kicks in because the >>> entire >>> >> connection is blocked, not just the one producer. There are two ways >>> >> this >>> >> can happen. First, if the client produces and consumes the same >>> >> destination. Second, if the client is producing one destination and >>> >> consuming a second destination and another client is consuming the >>> first >>> >> and >>> >> producing to the second. Actually, more complicated possibilities >>> exist >>> >> as >>> >> well. >>> >> >>> >> Your best bet is to never consume and produce on the same connection >>> so >>> >> that >>> >> consumption never blocks due to producer-flow-control. In that >>> >> scenario, >>> >> deadlocks won't happen because consumers can always consume. >>> >> >>> >> Back to stack traces - if the consuming threads are blocked waiting to >>> >> obtain a lock while producing threads hold that lock, that could >>> >> indicate >>> >> the problem. >>> >> >>> >> Anyway, slow consumption is quite often the cause of broker hangs, so >>> >> looking for slow consumption and understanding speeding consumption >>> (or >>> >> slowing production) is important. If the broker JVM becomes >>> completely >>> >> unresponsive, try connecting jconsole or visualvm before the broker >>> >> hangs >>> >> and then watch memory. If the JVM is running out of Heap or Permgen >>> >> space, >>> >> that would explain the hung broker. >>> >> >>> >> >>> >> >>> >> -- >>> >> View this message in context: >>> >> >>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html >>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >> >>> > >>> > >>> > >>> > >>> > _______________________________________________ >>> > If you reply to this email, your message will be added to the >>> discussion >>> > below: >>> > >>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html >>> > To start a new topic under ActiveMQ - User, email >>> > ml-node+s2283324n234180...@n4.nabble.com >>> > To unsubscribe from ActiveMQ - User, visit >>> > >>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== >>> >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> >
