What is the CPU load like on the broker host? Is that process spiking the CPU? If so, can you use JVisualVM or similar to do CPU sampling of the broker process and find out where it's spending its time?
My suspicion is that the broker is so busy that it's not responding to some connections fast enough to let them succeed. Tim On Wed, Nov 10, 2021, 2:17 AM Guillaume Cripiau < guillaume.crip...@murprogroup.com> wrote: > Hello, > > The only other logs (mqtt messages on the server) show this : > 10:30:45 > remove/ID:airprotec-cta-41267-1635489094200-3:8407/03200646/Connection > 10:30:46 > add/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection > 10:30:46 > remove/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection > > Yes, the client restarts after 2 hours without connection and the same > event happens but not always. > > The clients try to reconnect, the broker has just restarted. > > We cannot reproduce the problem on our dev server with minimal setup. > > We actually analyse the TCP connexion : the clients use the source port > 1883 and the timeout between a CONNECT command and CONNACK is 1 second. > Can this settings be the cause of our problems? > > Regards > > Guillaume Cripiau > > Le 02-11-21 à 12:45, Tim Bain a écrit : > > These are broker logs, or client logs? Whichever it is, what's in the > other > > process's logs at the same time? > > > > You said that reconnecting isn't possible for hours after this happens. > Do > > you see the same messages in the logs for that whole time? > > > > At the time this happens, what are the client(s) and the broker doing? > > > > Can you reproduce the problem reliably in a minimal setup (just one or > two > > clients, minimal time and message load)? > > > > Tim > > > > On Tue, Nov 2, 2021, 5:19 AM Guillaume Cripiau < > > guillaume.crip...@murprogroup.com> wrote: > > > >> Hello, > >> > >> We need support. Many devices disconnect for no apparent reason and > >> cannot reconnect after several days. Could you help us solve this > >> problem? Here is an example in the logs : > >> > >>> 2021-11-02 10:30:44,413 | WARN | Transport Connection to: > >>> tcp://62.83.10.35:1024 failed | > >>> org.apache.activemq.broker.TransportConnection.Transport | Async > >>> Exception Handler | localhost | | > >>> java.io.IOException: Connection reset > >>> at > >>> > >> > org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3] > >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > >>> Source)[:1.8.0_211] > >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > >>> Source)[:1.8.0_211] > >>> at java.lang.Thread.run(Unknown Source)[:1.8.0_211] > >>> 2021-11-02 10:30:46,359 | WARN | Transport Connection to: > >>> tcp://62.83.10.35:35598 failed | > >>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > >>> Transport: tcp:///62.83.10.35:35598@1883 | localhost | | > >>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Already > >>> connected. > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3] > >>> at java.lang.Thread.run(Unknown Source)[:1.8.0_211] > >>> 2021-11-02 10:31:19,226 | WARN | Transport Connection to: > >>> tcp://62.83.10.35:3000 failed | > >>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > >>> Transport: tcp:///62.83.10.35:3000@1883 | localhost | | > >>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Not > connected. > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3] > >>> at > >>> > >> > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3] > >>> at java.lang.Thread.run(Unknown Source)[:1.8.0_211] > >> Regards > >> > >> Guillaume Cripiau >
