aglmc4-git opened a new issue, #2787: URL: https://github.com/apache/activemq/issues/2787
**Environment** ActiveMQ Classic 5.19.11, Java 11, Linux About 290 static duplex SSL network connectors (static:(ssl://host:port)?useExponentialBackOff=false&initialReconnectDelay=5000&maxReconnectDelay=5000, duplex="true", staticBridge="true"), networkConnectorStartAsync="true", advisorySupport="false" About 15 of the remote hosts are permanently unreachable, and connection attempts to them hang No vm:// transport connector declared, so it's created automatically by the first network bridge The same configuration ran on 5.16.0 without this happening **Symptom** After about 8 days of uptime, every network bridge stopped within about one second and none reconnected. Nothing was logged at ERROR. Only the connectors to unreachable hosts kept retrying, because they were already in their reconnect loop. A broker restart fixed it. **Logging** at the moment of failure: 22:51:00 WARN Network connection between vm://localhost#1005752 and ssl://host/x.x.x.x:port shutdown due to a remote error: InactivityIOException: Channel was inactive for too (>30000) long 22:51:00 INFO localhost Shutting down <connector-1> 22:51:00 INFO localhost bridge to <host-1> stopped ... (~275 lines: "Shutting down <connector>" / "bridge to <host> stopped" for every connector, all in ~1s) 22:51:01 INFO Connector vm://localhost stopped 22:51:01 INFO Establishing network connection from vm://localhost to ssl://<unreachable-host>:port 22:51:01 INFO Connector vm://localhost started **AI Analysis (5.19.11 source)** The automatically created VM server is bound with disposeOnDisconnect=true (VMTransportFactory.doCompositeConnect: bind(location, true)). When its counter reaches 0, VMTransportFactory.stopped() disposes the TransportConnector and closes every VM connection. The client-side override in VMTransportServer.connect() decrements connectionCount on every stop() call. It guards on the server's disposed flag, not the transport's, so a second stop() on the same transport decrements again. 5.19.11 added VMTransport.stop(Throwable) (#2475), and stop() now delegates to it. The override only covers stop(), so a direct stop(Throwable) call skips the decrement. When the connector is disposed, each bridge's local side receives ShutdownInfo with error == null. DemandForwardingBridgeSupport.serviceLocalCommand then logs Shutting down and calls stop(), so no reconnect is triggered Evidence of drift (heap dumps; VMTransportServer.connectionCount.value): About 20 h after restart: 510 About 44 h after restart: 479 At the same time, 272 bridges were connected. Each duplex bridge holds 2 VM connections (localBroker plus duplexInboundLocalBroker), so the correct value would be about 544. Drift is about 1.3–1.7 per hour, consistent with reaching 0 after about 8 days. Suspected trigger (not confirmed) The failed-start path for unreachable hosts: Error with pending remote brokerInfo on: ssl://null:0, then Could not start network bridge ... Socket closed. On this path the same local transport can be stopped both by DemandForwardingBridgeSupport.stop() (ss.stop(localBroker)) and by DiscoveryNetworkConnector (ServiceSupport.dispose(localTransport)). Production logs show both orderings. The drift rate (~1.5/h) against roughly 1,500 failed attempts per hour suggests a race, not something that happens on every failure. Repeatedly pausing a remote broker with docker pause did not reproduce it in a test environment. **Workaround** Declare transportConnector name="vm" uri="vm://localhost" in the transport section of activemq.xml. It's bound with disposeOnDisconnect=false. The counter has since gone below 0, and all bridges stayed connected. **Suggested fix** Make the decrement idempotent per transport, with a per-instance guard in the client override. Cover stop(Throwable) in the override. Consider treating a VM connector disposal as an error for network bridges, so they reconnect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] For further information, visit: https://activemq.apache.org/contact
