Terje Strand created AMQ-7424:
---------------------------------
Summary: NPE under very high database load
Key: AMQ-7424
URL: https://issues.apache.org/jira/browse/AMQ-7424
Project: ActiveMQ
Issue Type: Bug
Components: JDBC
Affects Versions: 5.15.11
Reporter: Terje Strand
Under abnormally heavy database loads we get a lot of transactions timeouts in
our application as one would expect. Our application uses XA with Postgres and
ActiveMQ. Problem is that after the abnormality goes away, the system does not
recover.
During these failures, we get a NPE that causes ActiveMQ to lose a database
connection and the connection is never returned to the connection pool
(Hikari). After the abnormality is removed, and the database is responsive
again, the system never recovers as the connection pool is out-of-resources.
Through debugging, we believe the following causes the connection leak in
ActiveMQs handing:
{quote}Caused by: javax.jms.JMSException: java.lang.NullPointerException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1403)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1436)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.TransactionContext.rollback(TransactionContext.java:538)
~[activemq-client-5.15.11.jar:5.15.11]
... 134 more
Caused by: java.lang.NullPointerException
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.commitRemove(JDBCPersistenceAdapter.java:795)
~[activemq-jdbc-store-5.15.11.jar:5.15.11]
at
org.apache.activemq.store.jdbc.JdbcMemoryTransactionStore.rollback(JdbcMemoryTransactionStore.java:171)
~[activemq-jdbc-store-5.15.11.jar:5.15.11]
at
org.apache.activemq.transaction.XATransaction.rollback(XATransaction.java:146)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransactionBroker.rollbackTransaction(TransactionBroker.java:257)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection.processRollbackTransaction(TransportConnection.java:553)
~[activemq-broker-5.15.11.jar:5.15.11]
at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:104)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
~[activemq-client-5.15.11.jar:5.15.11]
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
~[activemq-client-5.15.11.jar:5.15.11]
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
~[activemq-client-5.15.11.jar:5.15.11]
... 1 more
{quote}
By overloading the method 'commitRemoved(...)' in 'JDBCPersistenceAdapter' and
converting the NullPointerException above to an IOException, the connection
handling code behaves as expected, we see no connection leak, and the system
recovers correctly after the load abnormality has passed.
There is a very large number of things going wrong when these NPEs occur and
its near impossible for us (not being experts at ActiveMQ) to see what the
underlying cause for these exceptions are. However, for us, the most important
is that we recover-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
