Bryan I haven't seen failover exhibit this problem before. The reconnection logic awaits the dispatcher (a session scoped thread which invokes your message listen) to finish delivery of the last message to your application before completing the reconnection. The dispatcher close allows a period of 1000ms for the dispatcher to complete its work before timing out. You could try increasing the timeout (system property -DDISPATCHER_SHUTDOWN_TIMEOUT_MS=<value in milliseconds>). However, as you say you message listener does not more than log a message I suspect there might be something else going on. If increasing the timeout doesn't help, share a complete log captured at DEBUG level and I'll try and help further.
HTH Keith. On 24 January 2018 at 18:44, bryand <[email protected]> wrote: > I'm using broker-j-7.0.0 and apache-qpid-jms-0-x-6.3.0. For broker-j I have > HA setup with 3 brokers and have been testing failover by simply starting > and stopping the virtual host node on a random broker. Usually failover on > my simple JMS message consumer is successful and message consumption > continues but occasionally I'll receive the following exception on my > consumer and it no longer receives messages: > java.lang.RuntimeException: Dispatcher did not close down within the timeout > of 1000 ms. > > My simple java client code is this.. > > String brokerUrl = > "amqp:///spgqpiddev?failover='roundrobin?cyclecount='2''&brokerlist='tcp://spgappdevmutil:5672?retries='3'&connectdelay='1000';tcp://appprd02:5672?retries='3'&connectdelay='1000';tcp://appprd02:5682?retries='3'&connectdelay='1000''"; > > ConnectionFactory connectionFactory = new AMQConnectionFactory(brokerUrl); > > String user = "appuser"; > String pwd = "xxxx"; > > Connection connection = connectionFactory.createConnection(user, pwd); > > connection.start(); > > Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE); > > String queueName = "app_testcluster"; > > Destination destination = AMQDestination.createDestination(queueName, > false); > > MessageConsumer messageConsumer = session.createConsumer(destination); > // this MsgListener class simply prints out messages received. > messageConsumer.setMessageListener(new MsgListener()); > > The full stacktrace is: > count: 500, id: ID:6c8f4df0-c8df-3751-86ee-75cef258de69, ts: 2018-01-24 > 09:58:49.718, lastTimestamp: 1516805929718, body: null > 2018-01-24 13:23:58,820 [10.0.51.42:5672] - ERROR AMQConnectionDelegate_0_10 > - error during failover > org.apache.qpid.transport.ConnectionException: connection closed > at org.apache.qpid.transport.Connection.send(Connection.java:414) > at org.apache.qpid.transport.Session.send(Session.java:588) > at org.apache.qpid.transport.Session.invoke(Session.java:804) > at org.apache.qpid.transport.Session.invoke(Session.java:613) > at > org.apache.qpid.transport.SessionInvoker.sessionAttach(SessionInvoker.java:29) > at org.apache.qpid.transport.Session.attach(Session.java:290) > at org.apache.qpid.transport.Session.resume(Session.java:300) > at org.apache.qpid.transport.Connection.resume(Connection.java:524) > at > org.apache.qpid.client.AMQConnectionDelegate_0_10.resubscribeSessions(AMQConnectionDelegate_0_10.java:288) > at > org.apache.qpid.client.AMQConnection.resubscribeSessions(AMQConnection.java:1480) > at > org.apache.qpid.client.AMQConnectionDelegate_0_10$2.run(AMQConnectionDelegate_0_10.java:363) > at > org.apache.qpid.client.AMQConnection.doWithAllLocks(AMQConnection.java:1959) > at > org.apache.qpid.client.AMQConnection.doWithAllLocks(AMQConnection.java:1947) > at > org.apache.qpid.client.AMQConnection.doWithAllLocks(AMQConnection.java:1926) > at > org.apache.qpid.client.AMQConnectionDelegate_0_10.closed(AMQConnectionDelegate_0_10.java:340) > at org.apache.qpid.transport.Connection.closed(Connection.java:601) > at > org.apache.qpid.transport.network.Assembler.closed(Assembler.java:113) > at > org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:219) > at > org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:225) > at java.lang.Thread.run(Thread.java:745) > 2018-01-24 13:23:58,822 [10.1.1.119:5682] - INFO FailoverRoundRobinServers > - ==== Checking failoverAllowed() ==== > 2018-01-24 13:23:58,822 [10.1.1.119:5682] - INFO FailoverRoundRobinServers > - Cycle Servers: > Cycle Retries:2 > Current Cycle:0 > Server Retries:3 > Current Retry:0 > Current Broker:2 > tcp://spgappdevmutil:5672?connectdelay='1000'&retries='3' > tcp://appprd02:5672?connectdelay='1000'&retries='3' >>tcp://appprd02:5682?connectdelay='1000'&retries='3' > > 2018-01-24 13:23:58,822 [10.1.1.119:5682] - INFO FailoverRoundRobinServers > - ==================================== > 2018-01-24 13:23:58,822 [10.1.1.119:5682] - INFO FailoverRoundRobinServers > - Trying tcp://appprd02:5682?connectdelay='1000'&retries='3' > 2018-01-24 13:23:58,823 [10.1.1.119:5682] - INFO FailoverRoundRobinServers > - Delay between connect retries:1000 > 2018-01-24 13:23:59,826 [10.0.51.42:5672] - ERROR > oggingUncaughtExceptionHandler - Uncaught exception in thread > "IoRcvr-/10.240.15.7:65068-spgappdevmutil/10.0.51.42:5672" > java.lang.RuntimeException: Dispatcher did not close down within the timeout > of 1000 ms. > at > org.apache.qpid.client.AMQSession$Dispatcher.close(AMQSession.java:3407) > at > org.apache.qpid.client.AMQSession.markClosedConsumers(AMQSession.java:3047) > at > org.apache.qpid.client.AMQSession.markClosedProducersAndConsumers(AMQSession.java:3077) > at org.apache.qpid.client.AMQSession.markClosed(AMQSession.java:2278) > at > org.apache.qpid.client.AMQConnectionDelegate_0_10.closed(AMQConnectionDelegate_0_10.java:392) > at org.apache.qpid.transport.Connection.closed(Connection.java:601) > at > org.apache.qpid.transport.network.Assembler.closed(Assembler.java:113) > at > org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:219) > at > org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:225) > at java.lang.Thread.run(Thread.java:745) > Exception in thread > "IoRcvr-/10.240.15.7:65068-spgappdevmutil/10.0.51.42:5672" > java.lang.RuntimeException: Dispatcher did not close down within the timeout > of 1000 ms. > at > org.apache.qpid.client.AMQSession$Dispatcher.close(AMQSession.java:3407) > at > org.apache.qpid.client.AMQSession.markClosedConsumers(AMQSession.java:3047) > at > org.apache.qpid.client.AMQSession.markClosedProducersAndConsumers(AMQSession.java:3077) > at org.apache.qpid.client.AMQSession.markClosed(AMQSession.java:2278) > at > org.apache.qpid.client.AMQConnectionDelegate_0_10.closed(AMQConnectionDelegate_0_10.java:392) > at org.apache.qpid.transport.Connection.closed(Connection.java:601) > at > org.apache.qpid.transport.network.Assembler.closed(Assembler.java:113) > at > org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:219) > at > org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:225) > at java.lang.Thread.run(Thread.java:745) > 2018-01-24 13:24:00,198 [10.1.1.119:5682] - INFO AMQConnection > - Cannot connect to broker > (tcp://appprd02:5682?connectdelay='1000'&retries='3'): Virtual host > 'spgqpiddev' is not active:Unable to connect to broker at > tcp://appprd02:5682?connectdelay='1000'&retries='3' > 2018-01-24 13:24:00,198 [10.1.1.119:5682] - INFO FailoverRoundRobinServers > - ==== Checking failoverAllowed() ==== > 2018-01-24 13:24:00,198 [10.1.1.119:5682] - INFO FailoverRoundRobinServers > - Cycle Servers: > > > > > -- > Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
