The log lines I posted are from the master side. The slave side log messages
are also interesting:
INFO 2015.01.10 14:00:30,077
org.apache.activemq.network.DemandForwardingBridgeSupport - Network
connection between vm://BALASSIBroker#114 and tcp:///10.36.0.20:62000@40819
shutdown due to a local error:
org.apache.activemq.transport.InactivityIOException: Channel was inactive
for too (>30000) long: tcp://10.36.0.20:62000
INFO 2015.01.10 14:00:30,083
org.apache.activemq.network.DiscoveryNetworkConnector - Establishing network
connection from vm://BALASSIBroker?async=false&network=true to
tcp://10.36.0.20:62000
WARN 2015.01.10 14:00:30,185
org.apache.activemq.broker.TransportConnection - Failed to add Connection
BALASSIBroker->PERCMASTERBroker-45698-1418680167169-14:2
javax.jms.InvalidClientIDException: Broker: BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
at
org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:242)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:91)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:759)
at
org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:291)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:145)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:247)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO 2015.01.10 14:00:30,189
org.apache.activemq.network.DemandForwardingBridgeSupport - Network
connection between vm://BALASSIBroker#118 and tcp:///10.36.0.20:62000@54620
shutdown due to a local error: javax.jms.InvalidClientIDException: Broker:
BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
INFO 2015.01.10 14:00:30,197
org.apache.activemq.network.DemandForwardingBridgeSupport - BALASSIBroker
bridge to PERCMASTERBroker stopped
INFO 2015.01.10 14:00:31,192
org.apache.activemq.network.DiscoveryNetworkConnector - Establishing network
connection from vm://BALASSIBroker?async=false&network=true to
tcp://10.36.0.20:62000
WARN 2015.01.10 14:00:31,266
org.apache.activemq.broker.TransportConnection - Failed to add Connection
BALASSIBroker->PERCMASTERBroker-45698-1418680167169-15:2
javax.jms.InvalidClientIDException: Broker: BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
at
org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:242)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:91)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:759)
at
org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:291)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:145)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:247)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO 2015.01.10 14:00:31,273
org.apache.activemq.network.DemandForwardingBridgeSupport - Network
connection between vm://BALASSIBroker#122 and tcp:///10.36.0.20:62000@54621
shutdown due to a local error: javax.jms.InvalidClientIDException: Broker:
BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
INFO 2015.01.10 14:00:31,284
org.apache.activemq.network.DemandForwardingBridgeSupport - BALASSIBroker
bridge to PERCMASTERBroker stopped
INFO 2015.01.10 14:00:33,276
org.apache.activemq.network.DiscoveryNetworkConnector - Establishing network
connection from vm://BALASSIBroker?async=false&network=true to
tcp://10.36.0.20:62000
WARN 2015.01.10 14:00:33,345
org.apache.activemq.broker.TransportConnection - Failed to add Connection
BALASSIBroker->PERCMASTERBroker-45698-1418680167169-16:2
javax.jms.InvalidClientIDException: Broker: BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
at
org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:242)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:91)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:759)
at
org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:291)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:145)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:247)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO 2015.01.10 14:00:33,348
org.apache.activemq.network.DemandForwardingBridgeSupport - Network
connection between vm://BALASSIBroker#126 and tcp:///10.36.0.20:62000@54622
shutdown due to a local error: javax.jms.InvalidClientIDException: Broker:
BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
INFO 2015.01.10 14:00:33,356
org.apache.activemq.network.DemandForwardingBridgeSupport - BALASSIBroker
bridge to PERCMASTERBroker stopped
INFO 2015.01.10 14:00:37,352
org.apache.activemq.network.DiscoveryNetworkConnector - Establishing network
connection from vm://BALASSIBroker?async=false&network=true to
tcp://10.36.0.20:62000
WARN 2015.01.10 14:00:37,426
org.apache.activemq.broker.TransportConnection - Failed to add Connection
BALASSIBroker->PERCMASTERBroker-45698-1418680167169-17:2
javax.jms.InvalidClientIDException: Broker: BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
at
org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:242)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:91)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
at
org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:759)
at
org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:291)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:145)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:247)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO 2015.01.10 14:00:37,428
org.apache.activemq.network.DemandForwardingBridgeSupport - Network
connection between vm://BALASSIBroker#130 and tcp:///10.36.0.20:62000@54624
shutdown due to a local error: javax.jms.InvalidClientIDException: Broker:
BALASSIBroker - Client:
clientConnector_PERCMASTERBroker_inbound_duplex_BALASSIBroker already
connected from vm://BALASSIBroker#116
INFO 2015.01.10 14:00:37,436
org.apache.activemq.network.DemandForwardingBridgeSupport - BALASSIBroker
bridge to PERCMASTERBroker stopped
INFO 2015.01.10 14:00:40,084
org.apache.activemq.network.DemandForwardingBridgeSupport - Network Could
not shutdown in a timely manner
WARN 2015.01.10 14:00:40,091
org.apache.activemq.network.DemandForwardingBridgeSupport - Caught an
exception processing local command
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.write(TcpBufferedOutputStream.java:96)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightMarshalByteSequence2(BaseDataStreamMarshaller.java:431)
at
org.apache.activemq.openwire.v10.MessageMarshaller.tightMarshal2(MessageMarshaller.java:182)
at
org.apache.activemq.openwire.v10.ActiveMQMessageMarshaller.tightMarshal2(ActiveMQMessageMarshaller.java:89)
at
org.apache.activemq.openwire.v10.ActiveMQTextMessageMarshaller.tightMarshal2(ActiveMQTextMessageMarshaller.java:89)
at
org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:243)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:175)
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:304)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:968)
at
org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:177)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
at
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
at
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1365)
at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:884)
at
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:930)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO 2015.01.10 14:00:40,095 org.apache.activemq.broker.TransportConnector
- Connector vm://BALASSIBroker stopped
INFO 2015.01.10 14:00:40,095
org.apache.activemq.network.DemandForwardingBridgeSupport - BALASSIBroker
bridge to PERCMASTERBroker stopped
INFO 2015.01.10 14:00:45,432
org.apache.activemq.network.DiscoveryNetworkConnector - Establishing network
connection from vm://BALASSIBroker?async=false&network=true to
tcp://10.36.0.20:62000
INFO 2015.01.10 14:00:45,433 org.apache.activemq.broker.TransportConnector
- Connector vm://BALASSIBroker started
INFO 2015.01.10 14:00:45,505
org.apache.activemq.network.DemandForwardingBridgeSupport - Network
connection between vm://BALASSIBroker#134 and tcp:///10.36.0.20:62000@54625
(PERCMASTERBroker) has been established.
WARN 2015.01.10 14:01:06,015
org.apache.activemq.broker.region.cursors.AbstractStoreCursor -
org.apache.activemq.broker.region.cursors.QueueStorePrefetch@61ee20ea:Consumer.BALASSI.VirtualTopic.SFOperationShare,batchResetNeeded=false,storeHasMessages=true,size=0,cacheEnabled=true,maxBatchSize:1,hasSpace:true
- cursor got duplicate:
ID:Perc-Smartfront1-33556-1418679846629-5:1:2:1:19939, 4
WARN 2015.01.10 14:01:06,017 org.apache.activemq.broker.region.Queue -
duplicate message from store
ID:Perc-Smartfront1-33556-1418679846629-5:1:2:1:19939, redirecting for dlq
processing
WARN 2015.01.10 14:01:06,071
org.apache.activemq.broker.region.cursors.AbstractStoreCursor -
org.apache.activemq.broker.region.cursors.QueueStorePrefetch@61ee20ea:Consumer.BALASSI.VirtualTopic.SFOperationShare,batchResetNeeded=false,storeHasMessages=false,size=1,cacheEnabled=true,maxBatchSize:1,hasSpace:true
- cursor got duplicate:
ID:Perc-Smartfront1-33556-1418679846629-5:1:17:1:14867, 4
WARN 2015.01.10 14:01:06,072 org.apache.activemq.broker.region.Queue -
duplicate message from store
ID:Perc-Smartfront1-33556-1418679846629-5:1:17:1:14867, redirecting for dlq
processing
Thanks,
Zoltan
--
View this message in context:
http://activemq.2283324.n4.nabble.com/ActiveMQ-embedded-broker-100-cpu-usage-tp4689765p4689774.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
