Hello,
I am facing a weird connection reset errors being logged at debug level
randomly. I am using ACTIVEMQ 5.14.4 with JDK 8-141 deployed on a Windows
2012 R2 Machine. The ActiveMQ Broker has 4 GB heap defined with almost 14000
connections (Yeah we havent used any pooling yet it uses a different
connection for every queue/topic per thread) for queues and topics across
various java processes.
The message seen is logs are :-
07:09:55:740|1209-00246:DEBUG
[org.apache.activemq.broker.TransportConnection.Transport] - Transport
Connection to: tcp://172.31.26.125:50898 failed: java.net.SocketException:
Connection reset Thread {ActiveMQ Transport:
tcp:///172.31.26.125:50898@61616}
java.net.SocketException: Connection reset
at
java.net.SocketInputStream.read(SocketInputStream.java:210)
at
java.net.SocketInputStream.read(SocketInputStream.java:141)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:748)
07:09:55:740|0324-00246:DEBUG
[org.apache.activemq.broker.jmx.ManagementContext] - Unregistering MBean
org.apache.activemq:type=Broker,brokerName=AMQ,connector=clientConnectors,connectorName=tcp,connectionViewType=clientId,connectionName=ID_WIN-31THCOHACHB-49421-1509705222509-4_1
Thread {ActiveMQ Transport: tcp:///172.31.26.125:50898@61616}
07:09:55:740|0312-00246:DEBUG
[org.apache.activemq.broker.jmx.ManagementContext] - Unregistering MBean
org.apache.activemq:type=Broker,brokerName=AMQ,connector=clientConnectors,connectorName=tcp,connectionViewType=remoteAddress,connectionName=tcp_//172.31.26.125_50898
Thread {ActiveMQ Transport: tcp:///172.31.26.125:50898@61616}
07:09:55:740|0155-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Stopping connection:
tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ] Task-103}
07:09:55:740|0160-00675:DEBUG
[org.apache.activemq.transport.tcp.TcpTransport] - Stopping transport
tcp:///172.31.26.125:50898@61616 Thread {ActiveMQ BrokerService[AMQ]
Task-103}
07:09:55:740|0306-00675:DEBUG [org.apache.activemq.thread.TaskRunnerFactory]
- Initialized TaskRunnerFactory[ActiveMQ Task] using ExecutorService:
java.util.concurrent.ThreadPoolExecutor@30b3bb2e[Running, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0] Thread {ActiveMQ
BrokerService[AMQ] Task-103}
07:09:55:741|0156-00714:DEBUG
[org.apache.activemq.transport.tcp.TcpTransport] - Closed socket
Socket[addr=/172.31.26.125,port=50898,localport=61616] Thread {ActiveMQ
Task-1}
07:09:55:741|0271-00675:DEBUG [org.apache.activemq.util.ThreadPoolUtils] -
Forcing shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@30b3bb2e[Running, pool size = 1,
active threads = 0, queued tasks = 0, completed tasks = 1] Thread {ActiveMQ
BrokerService[AMQ] Task-103}
07:09:55:741|0153-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Stopped transport:
tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ] Task-103}
07:09:55:741|0168-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Cleaning up connection
resources: tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ]
Task-103}
07:09:55:741|0173-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - remove connection id:
ID:WIN-31THCOHACHB-49421-1509705222509-5:1 Thread {ActiveMQ
BrokerService[AMQ] Task-103}
Things that i have tried
1. Setting TCP level Keep Alive to true
2. Disabling InactivityMonitorThread by setting Max Inactivity Duration
to 0
3. using connection pool for creating connections which brought the
entire connection count to 300-350 connections in all, thus reducing load on
OS
But none of the above changes have worked for me to get rid of the problem.
my entire infrastructure is hosted on AWS and CPU usage not going beyond
15-20%. Also I am not using failover protocol
I would like to know 1) are these exception traces considered to be harmful
? 2) If yes what could be causing this connection reset ?
Let me know if any more info is needed.
Regards,
-Yogesh
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
