Re: JMS exception during the Failover
Tim , Thank you for your suggestion. But i have tried different way now to see the result. It worked out. I have created a new maven project with only producer and consumer and the old project contains unnecessary dependencies which might have caused the issue. As soon as i have tried with the new project with just activemq core dependency the failover started working. It might be related to the dependency or manifest of the project which got corrupted. So finally , there are no issues in regards to the failover at this moment. So it's better always whenever we are making trials with failover it's better we are handling the project manifest and dependencies in a right way. Thank you very much for your suggestion. -- View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4717069.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: JMS exception during the Failover
If you switch your acknowledgement mode to AUTO_ACKNOWLEDGE and comment out your call to message.acknowledge(), does the behavior change? Client ack mode isn't very widely used, so it's possible there's a bug in the interaction between it and failover. Also, in your original post, you said that threads were blocked, and provided a screenshot to illustrate why you thought that. That thread is an idle thread in a thread pool, and it's blocked because there's not enough work to keep all the threads busy, not because anything is wrong or because ActiveMQ threads are being prevented from running. Tim On Sep 21, 2016 10:33 AM, "akhil" <akh...@gmail.com> wrote: > Hi Tim , > > I am sorry that one was the outdated consumer , I have changed it long time > back and forgot to update the gist. Yes i have tried yourway but the same > result. > > Thanks, > Akhil. > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/JMS-exception-during-the-Failover-tp4716047p4716706.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
Re: JMS exception during the Failover
Hi Tim , I am sorry that one was the outdated consumer , I have changed it long time back and forgot to update the gist. Yes i have tried yourway but the same result. Thanks, Akhil. -- View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716706.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: JMS exception during the Failover
The catch block in your consumer looks very suspicious, since it starts additional consumers without closing the previous ones. Would you please comment out the line that's there, and also add a line to log the exception you're catching? Tim On Sep 20, 2016 11:16 AM, "akhil" <akh...@gmail.com> wrote: > Hi Tim , > > I have tried with the graceful shutdown by not using kill any more but > still > it's on the same way of dealing the consumer thread block. > > you can find the new switch log here : > https://gist.github.com/areddy7021/e0e3a6c89fc974ce3031e326c060acca > > I have switched like this way ..started with Broker A as master and B as > slave ..-- > everything is fine > > Broker A service down and Broker B as new master -- > Producer is fine and > Consumer started throwing exception and still consumed even after the > switch > since some buffered messages in prefetch buffer and after that inactivity > monitor started throwing up and connection state in in blocked state in > jconsole. > > Broker B service down and Broker A as new master ---> Producer no issues as > usual. Consumer started showing some log in attempt to connect to new > master > but its not done with the inactivity log. The above gist log is for this > scenario. > > Thanks, > Akhil. > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/JMS-exception-during-the-Failover-tp4716047p4716660.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
Re: JMS exception during the Failover
Hi Tim , I have tried with the graceful shutdown by not using kill any more but still it's on the same way of dealing the consumer thread block. you can find the new switch log here : https://gist.github.com/areddy7021/e0e3a6c89fc974ce3031e326c060acca I have switched like this way ..started with Broker A as master and B as slave ..-- > everything is fine Broker A service down and Broker B as new master -- > Producer is fine and Consumer started throwing exception and still consumed even after the switch since some buffered messages in prefetch buffer and after that inactivity monitor started throwing up and connection state in in blocked state in jconsole. Broker B service down and Broker A as new master ---> Producer no issues as usual. Consumer started showing some log in attempt to connect to new master but its not done with the inactivity log. The above gist log is for this scenario. Thanks, Akhil. -- View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716660.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: JMS exception during the Failover
Thank You Tim ..I am just doing kill now on the activemq process instead of service shut down. I can try the service shutdown as per your suggestion. When i get into this state like the consumer idle or inactivity state ..from consumer point there is no activity though we switch broker from B to A. Yes it is consistently reproducible. Here is the Consumer Log where you can find the inactivity monitor. https://gist.github.com/areddy7021/da6800b18bb4118e229e77c7c2fcc422 Here is the consumer : https://gist.github.com/areddy7021/f3bdd4b30e656b3bcab03a0bec31da99 The consumer connection block : <http://activemq.2283324.n4.nabble.com/file/n4716652/efs.png> Thanks, Akhil. -- View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716652.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: JMS exception during the Failover
Am I right in understanding that you're no longer having the problem from your first email, and the problem from yesterday is the only one? When you stop the broker, how are you stopping it? If you're doing kill -9, that doesn't gracefully shut down the TCP connections, so the behavior can be different than if you issue a stop at the command prompt. It should still work eventually (and since it's not working for you, then it's either a bug in our code or a bug in yours), but be aware that the clean shutdown case will be faster than the hard kill case, so don't expect identical behavior between the two cases. Can you post your consumer configuration/code so we can see if there is something obvious that might explain what you're seeing? Also, it sounds like you took a thread dump when this was happening; are you able to post it? Also, is this consistently reproducible? And what happens if you start A and stop B once you're in this state? Tim On Sep 19, 2016 6:25 PM, "akhil" <akh...@gmail.com> wrote: > Hi , I am having the same type of issue but with the different > configuration. > I have two brokers which are not in network but i am using producer and > consumer with two different threads in my local and trying to hit 10k > messages and consume it from second thread. I am using the shared file > storage mount for the kaha db mount.One of the broker is acting as a master > any any point of time and the other one is slave. I have started my test > with Broker A as master and producer is sending out messages and consumer > started listening to it.. to add the complexity i have turned of the Broker > A and the lock got acquired by Broker B and it is now new master. Producer > got reconnected to master but consumer is failing and invoked the > inactivity > monitor and i have observed the TIME_WAIT thread block of connection broker > of consumer thread and it never ever reconnected to the new master ? > > Producer no issues in regards to the reconnection in failover scenario but > consumer does ..any one any suggestions ..is it something real bug with a > race condition or some thing we need to put it in the consumer code to > avoid > this failure ? > > Thanks, > Akhil. > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/JMS-exception-during-the-Failover-tp4716047p4716642.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
Re: JMS exception during the Failover
Hi , I am having the same type of issue but with the different configuration. I have two brokers which are not in network but i am using producer and consumer with two different threads in my local and trying to hit 10k messages and consume it from second thread. I am using the shared file storage mount for the kaha db mount.One of the broker is acting as a master any any point of time and the other one is slave. I have started my test with Broker A as master and producer is sending out messages and consumer started listening to it.. to add the complexity i have turned of the Broker A and the lock got acquired by Broker B and it is now new master. Producer got reconnected to master but consumer is failing and invoked the inactivity monitor and i have observed the TIME_WAIT thread block of connection broker of consumer thread and it never ever reconnected to the new master ? Producer no issues in regards to the reconnection in failover scenario but consumer does ..any one any suggestions ..is it something real bug with a race condition or some thing we need to put it in the consumer code to avoid this failure ? Thanks, Akhil. -- View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047p4716642.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
JMS exception during the Failover
I am enabling the shared file storage for two independent brokers with the EFS mount. While i am testing the shared file storage with the two individual brokers , one of the master broker is acquiring the lock and it's working as expected in case of publishing and consuming. The client is using the failover url for the two individual brokers to get the connection object. Now i have put the master broker down and with in short span experiencing the following log in the consumer thread log. In case of publisher thread there is no unusual behaviour in publishing the messages to the new failover broker which became master. And i see that consumer is not working in regards to consuming the messages from the new master broker. Is this the expected behaviour in case of failover where consumer is going to be offline and publisher is not even impacted. - Async error occurred: javax.jms.JMSException: Unmatched acknowledege: > MessageAck {commandId = 47577, responseRequired = false, ackType = 2, > consumerId = -- View this message in context: http://activemq.2283324.n4.nabble.com/JMS-exception-during-the-Failover-tp4716047.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.