[
https://issues.apache.org/jira/browse/AMQ-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JJ updated AMQ-7229:
--------------------
Summary: JmsConnector handling loss of connection [ActiveMQConnection
{id=ID:xs-38677-1560088234347-2:100,clientId=ID:xs.test.int-38677-1560088234347-1:100,started=true}]
| org.apache.activemq.network.jms.JmsConnector | ActiveMQ Session Task-8514
(was: JmsConnector handling loss of connection [ActiveMQConnection
{id=ID:xs-38677-1560088234347-2:100,clientId=ID:vm1.qrail.uk-38677-1560088234347-1:100,started=true}]
| org.apache.activemq.network.jms.JmsConnector | ActiveMQ Session Task-8514)
> JmsConnector handling loss of connection [ActiveMQConnection
> {id=ID:xs-38677-1560088234347-2:100,clientId=ID:xs.test.int-38677-1560088234347-1:100,started=true}]
> | org.apache.activemq.network.jms.JmsConnector | ActiveMQ Session Task-8514
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-7229
> URL: https://issues.apache.org/jira/browse/AMQ-7229
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, Connector, JMS client
> Affects Versions: 5.15.8, 5.15.9
> Environment: CentOS 6 & 7
> Default JARS
> Reporter: JJ
> Priority: Blocker
> Labels: beginner, newbie
>
> Periodically I am getting the following logged to my production and dev AMQ
> instances;
> JmsConnector handling loss of connection [ActiveMQConnection
> \\{id=ID:server-38677-1560088234347-2:100,clientId=ID:vm1.test.int-38677-1560088234347-1:100,started=true}]
> | org.apache.activemq.network.jms.JmsConnector | ActiveMQ Session Task-8514
>
> This come at random times from two separate servers on separate networks
> (connecting to a common upstream provider) These errors ALWAYS come in groups
> of 11, with the time spanning 11 sec - both servers see the same error at the
> same time (Indicating something at the far end maybe?)
> When I check the connection count in a JMX console it has increased by 11,
> with none of the old connections seeming to close or get cleaned up.
> System config:
> JMS-JMS Bridge with embedded broker bridging topics from a remote server to
> local server. Local clients connect with nio+stomp and all have persistent
> subscriptions to topics.
> The remote connection is openwire.
> When this error occurs it leads to a corrupted message which then causes the
> DB store to grow until an OOM error occurs and AMQ stops processing. The only
> way to resolve the issue is to stop all the connected clients and clear there
> subscriptions and then reconnect.
> I have done wireshark on a dev broker but can't see anything *obvious* though
> wireshark doesn't like to dissect openwire so I can't see what's actually
> going on.
> This error sometimes occurs days apart, or like today multiple times a day.
>
> The error logged in debug is:
> 2019-04-02 19:29:20,360 | TRACE | ID:xs-39837-1553964750839-4:140:2 sending
> message: ActiveMQTextMessage {commandId = 3319310, responseRequired = false,
> mes
> sageId = ID:xs-39837-1553964750839-4:140:2:1:1, originalDestination = null,
> originalTransactionId = null, producerId =
> ID:xs-39837-1553964750839-4:140:2:1,
> destination = topic://TD_ALL, transactionId = null, expiration = 0,
> timestamp = 1554229760360, arrival = 0, brokerInTime = 1554229755115,
> brokerOutTime = 15
> 54229755118, correlationId = null, replyTo = null, persistent = true, type =
> null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId =
> null,
> compressed = true, userID = null, content =
> org.apache.activemq.util.ByteSequence@2a02b7e9, marshalledProperties =
> org.apache.activemq.util.ByteSequence@7d
> db0ca1, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
> null, readOnlyProperties = true, readOnlyBody = true, droppable = false,
> jmsXGroupFirstForConsumer = false, text =
> [\\{"SF_MSG":{"time":"1554229754000","area_id":...r":"2W58"}}]} |
> org.apache.activemq.ActiveMQSession | ActiveMQ Session Task-4447
> 2019-04-02 19:29:20,360 | TRACE | Running task iteration 0 -
> vm://xs.test.int#279 | org.apache.activemq.thread.PooledTaskRunner | ActiveMQ
> VMTransport: vm://xs.test.int#279-1
> 2019-04-02 19:29:20,400 | DEBUG | Error occured while processing sync
> command: ActiveMQTextMessage {commandId = 7, responseRequired = true,
> messageId = ID:xs-39837-1553964750839-4:140:2:1:1, originalDestination =
> null, originalTransactionId = null, producerId =
> ID:xs-39837-1553964750839-4:140:2:1, destination = topic://TD_ALL,
> transactionId = null, expiration = 0, timestamp = 1554229760360, arrival = 0,
> brokerInTime = 1554229760361, brokerOutTime = 1554229755118, correlationId =
> null, replyTo = null, persistent = true, type = null, priority = 4, groupID =
> null, groupSequence = 0, targetConsumerId = null, compressed = true, userID =
> null, content = org.apache.activemq.util.ByteSequence@2a02b7e9,
> marshalledProperties = org.apache.activemq.util.ByteSequence@7ddb0ca1,
> dataStructure = null, redeliveryCounter = 0, size = 1595, properties = null,
> readOnlyProperties = true, readOnlyBody = true, droppable = false,
> jmsXGroupFirstForConsumer = false, text =
> [\{"SF_MSG":{"time":"1554229754000","area_id":...r":"2W58"}}]}, exception:
> java.io.EOFException | org.apache.activemq.broker.TransportConnection.Service
> | ActiveMQ VMTransport: vm://xs.test.int#279-1
> java.io.EOFException
> at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)[:1.8.0_201]
> at
> java.io.DataInputStream.readUTF(DataInputStream.java:589)[:1.8.0_201]
> at
> java.io.DataInputStream.readUTF(DataInputStream.java:564)[:1.8.0_201]
> at
> org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:97)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:78)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:70)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.command.Message.unmarsallProperties(Message.java:252)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.command.Message.getProperty(Message.java:202)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.region.policy.RetainedMessageSubscriptionRecoveryPolicy.add(RetainedMessageSubscriptionRecoveryPolicy.java:53)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:756)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:556)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.region.Topic.send(Topic.java:484)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:138)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:508)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:459)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:293)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:226)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:578)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:275)[activemq-broker-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.15.9.jar:5.15.9]
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.15.9.jar:5.15.9]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_201]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_201]
> at java.lang.Thread.run(Thread.java:748)[:1.8.0_201]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
