Tamas Szabo created AMQ-6609:
--------------------------------
Summary: A lot async error and null pointer exception
Key: AMQ-6609
URL: https://issues.apache.org/jira/browse/AMQ-6609
Project: ActiveMQ
Issue Type: Bug
Components: JMX, networkbridge, Transport
Affects Versions: 5.5.0
Environment: 2 virtual machines, in same network.
Having 2 AMQ server instance, with a network connection. HA - Active-Active
mode.
*CONFIG#1*
{code:xml}
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="xx1"
dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true"
deleteAllMessagesOnStartup="true" networkConnectorStartAsync="true"
enableStatistics="false" schedulerSupport="false" useJmx="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true"
memoryLimit="5mb">
<pendingSubscriberPolicy>
<fileCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true"
memoryLimit="5mb" optimizedDispatch="true">
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false" />
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"
enableIndexWriteAsync="true" />
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="500 mb" />
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="1 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
uri="nio://0.0.0.0:5672?keepAlive=true&soTimeout=60000&wireFormat.maxInactivityDurationInitalDelay=8000&wireFormat.maxInactivityDuration=19000"
/>
</transportConnectors>
</broker>
<import resource="jetty.xml" />
</beans>
{code}
*CONFIG#2:*
{code:xml}
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="xx2"
dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true"
deleteAllMessagesOnStartup="true" networkConnectorStartAsync="true"
enableStatistics="false" schedulerSupport="false" useJmx="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true"
memoryLimit="5mb">
<pendingSubscriberPolicy>
<fileCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true"
memoryLimit="5mb" optimizedDispatch="true">
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false" />
</managementContext>
<networkConnectors>
<networkConnector uri="static:(tcp://xx1:5672)" duplex="true"
conduitSubscriptions="true" decreaseNetworkConsumerPriority="true" />
</networkConnectors>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"
enableIndexWriteAsync="true" />
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="500 mb" />
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="1 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
uri="nio://0.0.0.0:5672?keepAlive=true&soTimeout=60000&wireFormat.maxInactivityDurationInitalDelay=8000&wireFormat.maxInactivityDuration=19000"
/>
</transportConnectors>
</broker>
<import resource="jetty.xml" />
</beans>
{code}
Reporter: Tamas Szabo
We have a production system installed, and on peaks we get a lot error because
of undelivered messages.
We have a java application which usually does the logic, and a c# application
which makes requests to the java app.
The error is customer sees a lot slowness in their system because of these
errors.
Errors are following:
*Error-1*
*XX-1 LOG*
{quote}
2017-02-21 08:07:06,075 | WARN | Network connection between vm://xx-2#68 and
tcp:///xx-2:5672 shutdown due to a remote error: java.net.SocketException:
Software caused connection abort: recv failed |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport:
tcp:///xx-2:5672
2017-02-21 08:07:06,075 | INFO | Network connection between vm://xx-2#68 and
tcp:///xx-2:5672 shutdown due to a local error: java.net.SocketException:
Software caused connection abort: socket write error |
org.apache.activemq.network.DemandForwardingBridgeSupport | BrokerService[xx-2]
Task-472152
{quote}
*XX-2 LOG*
{quote}
2017-02-21 08:07:05,476 | INFO | Network connection between vm://xx-1#68 and
tcp:///xx-1:50234 shutdown due to a local error: javax.jms.JMSException: The
destination temp-queue://ID:XYZA1234-65130-636232610113011707-1:0:2 does not
exist. | org.apache.activemq.network.DemandForwardingBridgeSupport |
BrokerService[xx-1] Task-592541
2017-02-21 08:07:10,500 | INFO | The connection to 'null' is taking a long
time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ
Task-15133
2017-02-21 08:07:15,508 | INFO | The connection to 'null' is taking a long
time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ
Task-15133
2017-02-21 08:07:20,520 | INFO | The connection to 'null' is taking a long
time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ
Task-15133
2017-02-21 08:07:25,528 | INFO | The connection to 'null' is taking a long
time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ
Task-15133
2017-02-21 08:07:30,536 | INFO | The connection to 'null' is taking a long
time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ
Task-15133
2017-02-21 08:07:30,941 | INFO | Connector vm://xx-1 Stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ Task-15134
2017-02-21 08:07:30,941 | INFO | xx-1 bridge to xx-2 stopped |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Task-15134
2017-02-21 08:08:11,051 | INFO | Connector vm://xx-1 Started |
org.apache.activemq.broker.TransportConnector | ActiveMQ NIO Worker
2017-02-21 08:08:11,083 | INFO | Started responder end of duplex bridge
NC@ID:XX-2-55636-1486253750115-0:1 |
org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker
2017-02-21 08:08:11,083 | INFO | Network connection between vm://xx-1#70 and
tcp:///xx-1:50249(xx-2) has been established. |
org.apache.activemq.network.DemandForwardingBridgeSupport | StartLocalBridge:
localBroker=vm://xx-1#70
2017-02-21 08:32:23,098 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 08:39:44,635 | INFO | Transport failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>19000) long: /11.111.111.11:64200 |
org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor
Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@37d226bd
2017-02-21 08:47:31,816 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 08:51:17,393 | INFO | Transport failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>19000) long: /11.111.111.12:53436 |
org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor
Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@511249ed
2017-02-21 08:51:47,940 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 08:59:18,723 | INFO | Transport failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>19000) long: /11.111.111.14:51599 |
org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor
Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@3d79ba00
2017-02-21 09:21:23,154 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 09:28:38,819 | INFO | Transport failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>19000) long: /11.111.111.14:49185 |
org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor
Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@6a3566bf
2017-02-21 09:33:57,336 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 09:49:52,090 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 09:54:22,246 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 10:12:02,324 | INFO | Transport failed: java.io.IOException: An
existing connection was forcibly closed by the remote host |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 11:14:57,879 | INFO | ActiveMQ Message Broker (xx-1,
ID:XX-1-49435-1486253561864-0:1) is shutting down |
org.apache.activemq.broker.BrokerService | Thread-24
2017-02-21 11:14:57,879 | INFO | ActiveMQ Message Broker (xx-1,
ID:XX-1-49435-1486253561864-0:1) is shutting down |
org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
{quote}
*Error-2*
We have a lot of following errors (daily hundreds or more.)
{quote}
2017-02-21 08:07:05,476 | WARN | Async error occurred: javax.jms.JMSException:
The destination temp-queue://ID:XYZA1234-65130-123123123123123123-1:0:2 does
not exist. | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ
NIO Worker
javax.jms.JMSException: The destination
temp-queue://ID:XYZA1234-65130-123123123123123123-1:0:2 does not exist.
at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:135)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
