have you peeked at https://issues.apache.org/jira/browse/AMQ-3451
On 19 February 2014 15:47, Chirag Pujara <chiragpuj...@gmail.com> wrote: > Update: > > I think I found problem why I see this behavior. We have bunch wars running > on tomcat and sending messages to broker. We have uptime deployment process > for all the wars. So when I undeploy war I think activemq connection > threads dont shut down properly and thats why I see exceptions like this in > broker logs: > > 2014-02-18 19:54:20,664 | WARN | Transport Connection to: tcp:// > 10.190.225.56:39399 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.56:39399@61618 > 2014-02-18 19:54:20,664 | WARN | Transport Connection to: tcp:// > 10.190.225.56:39412 failed: java.net.SocketException: Connection reset | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.56:39412@61618 > 2014-02-18 19:54:20,664 | WARN | Transport Connection to: tcp:// > 10.190.225.56:39409 failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///10.190.225.56:39409@61618 > 2014-02-18 19:55:42,270 | WARN | Transport Connection to: tcp:// > 10.190.133.15:33382 failed: > org.apache.activemq.transport.InactivityIOException: Channel was inactive > for too (>1000) long: tcp://10.190.133.15:33382 | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > InactivityMonitor Worker > > I have tried tcpProperty: *daemon=true* on service side and on client side. > But if I understand right it helps cleaning up threads when JVM shuts down. > In my case we dont stop tomcat. All our deployment happens in uptime. > > Anyother suggestion that I can try? > > thanks, > chirag > > > On Tue, Feb 11, 2014 at 4:46 PM, Chirag Pujara <chiragpuj...@gmail.com>wrote: > >> I tried again and this time I didnt send bunch of messages at same time. I >> noticed that some message gets processed right away but some messges stays >> on queue and doesnt get processed. If send few more messges let say 50 >> messages and it processes 40 messages and 10 msgs left on queue. I can send >> more messages but not all get processed. >> >> I am attaching config file. I have enabled producerFlowControl and removed >> slowConsumerStrategy. I am using activemq-5.10-SNAPSHOT. >> >> >> >> On Mon, Feb 10, 2014 at 9:04 PM, Chirag Pujara <chiragpuj...@gmail.com>wrote: >> >>> i said broker hangs bcs if i tried to send more messages or messages on >>> queue doent get processed. i cannot even create queue or send message using >>> webconsole. >>> >>> will try again and rhis time will moniter memory. >>> >>> On Feb 10, 2014 6:22 PM, "artnaseef" <a...@artnaseef.com> wrote:< >>>> >>>> Perhaps I misunderstood "Broker Hangs". It helps to have more detailed >>>> symptoms. >>>> >>>> The messages in the DLQ are failing at the consumer - either transactions >>>> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE >>>> client calling Session.recover(). >>>> >>>> See here for more details: >>>> http://activemq.apache.org/message-redelivery-and-dlq-handling.html >>>> >>>> You are right about the webconsole -- it will most likely stop >>>> functioning >>>> once the broker runs out of memory. >>>> >>>> > >>>> > >>>> > I have producerFlowControl to false. I seperated my producer and >>>> > consumers. >>>> > But still I was able to reproduce issue. For some time bunch of >>>> messages >>>> > started pile up in queue. Then they started to endup in DLQ. In DLQ I >>>> saw >>>> > most of them have these property: >>>> > >>>> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery >>>> policy >>>> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor = >>>> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1, >>>> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false, >>>> > useExponentialBackOff = false, backOffMultiplier = 5.0, >>>> redeliveryDelay = >>>> > 1000}, cause:null >>>> > >>>> > Does this thing has anything to do with the issue? >>>> > >>>> > I was not monitoring heap size when it got stuck. that will be next >>>> thing >>>> > to do. But I can hit webconsole from that activemq instance. So if it >>>> had >>>> > memory issues how come console is still working? isn't it same process? >>>> > >>>> > thanks, >>>> > chirag >>>> > >>>> > >>>> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <a...@artnaseef.com> wrote: >>>> > >>>> >> How about a stack trace on the client? Can you look for the consumer >>>> >> threads >>>> >> and see what they are doing? >>>> >> >>>> >> Note that with producers and consumers on the same connection, it's >>>> >> possible >>>> >> to reach a deadlock if producer-flow-control kicks in because the >>>> entire >>>> >> connection is blocked, not just the one producer. There are two ways >>>> >> this >>>> >> can happen. First, if the client produces and consumes the same >>>> >> destination. Second, if the client is producing one destination and >>>> >> consuming a second destination and another client is consuming the >>>> first >>>> >> and >>>> >> producing to the second. Actually, more complicated possibilities >>>> exist >>>> >> as >>>> >> well. >>>> >> >>>> >> Your best bet is to never consume and produce on the same connection >>>> so >>>> >> that >>>> >> consumption never blocks due to producer-flow-control. In that >>>> >> scenario, >>>> >> deadlocks won't happen because consumers can always consume. >>>> >> >>>> >> Back to stack traces - if the consuming threads are blocked waiting to >>>> >> obtain a lock while producing threads hold that lock, that could >>>> >> indicate >>>> >> the problem. >>>> >> >>>> >> Anyway, slow consumption is quite often the cause of broker hangs, so >>>> >> looking for slow consumption and understanding speeding consumption >>>> (or >>>> >> slowing production) is important. If the broker JVM becomes >>>> completely >>>> >> unresponsive, try connecting jconsole or visualvm before the broker >>>> >> hangs >>>> >> and then watch memory. If the JVM is running out of Heap or Permgen >>>> >> space, >>>> >> that would explain the hung broker. >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> View this message in context: >>>> >> >>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html >>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>> >> >>>> > >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > If you reply to this email, your message will be added to the >>>> discussion >>>> > below: >>>> > >>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html >>>> > To start a new topic under ActiveMQ - User, email >>>> > ml-node+s2283324n234180...@n4.nabble.com >>>> > To unsubscribe from ActiveMQ - User, visit >>>> > >>>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ== >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html >>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >>> >> -- http://redhat.com http://blog.garytully.com
