[
https://issues.apache.org/jira/browse/AMQ-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Bertram updated AMQ-7424:
--------------------------------
Description:
Under abnormally heavy database loads we get a lot of transactions timeouts in
our application as one would expect. Our application uses XA with Postgres and
ActiveMQ. The problem is that after the abnormality goes away the system does
not recover.
During these failures we get an NPE that causes ActiveMQ to lose a database
connection and the connection is never returned to the connection pool
(Hikari). After the abnormality is removed and the database is responsive again
the system never recovers as the connection pool is out-of-resources.
Through debugging, we believe the following causes the connection leak in
ActiveMQs handing:
{noformat}Caused by: javax.jms.JMSException: java.lang.NullPointerException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1403)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1436)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.TransactionContext.rollback(TransactionContext.java:538)
~[activemq-client-5.15.11.jar:5.15.11]
... 134 more
Caused by: java.lang.NullPointerException
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.commitRemove(JDBCPersistenceAdapter.java:795)
~[activemq-jdbc-store-5.15.11.jar:5.15.11]
at
org.apache.activemq.store.jdbc.JdbcMemoryTransactionStore.rollback(JdbcMemoryTransactionStore.java:171)
~[activemq-jdbc-store-5.15.11.jar:5.15.11]
at
org.apache.activemq.transaction.XATransaction.rollback(XATransaction.java:146)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransactionBroker.rollbackTransaction(TransactionBroker.java:257)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection.processRollbackTransaction(TransportConnection.java:553)
~[activemq-broker-5.15.11.jar:5.15.11]
at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:104)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
~[activemq-client-5.15.11.jar:5.15.11]
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
~[activemq-client-5.15.11.jar:5.15.11]
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
~[activemq-client-5.15.11.jar:5.15.11]
... 1 more{noformat}
By overloading the method {{commitRemoved(...)}} in {{JDBCPersistenceAdapter}}
and converting the {{NullPointerException}} above to an {{IOException}}, the
connection handling code behaves as expected. We see no connection leak, and
the system recovers correctly after the load abnormality has passed.
There is a very large number of things going wrong when these NPEs occur and
its near impossible for us (not being experts at ActiveMQ) to see what the
underlying cause for these exceptions are. However, for us, the most important
is that we recover.
was:
Under abnormally heavy database loads we get a lot of transactions timeouts in
our application as one would expect. Our application uses XA with Postgres and
ActiveMQ. Problem is that after the abnormality goes away, the system does not
recover.
During these failures, we get a NPE that causes ActiveMQ to lose a database
connection and the connection is never returned to the connection pool
(Hikari). After the abnormality is removed, and the database is responsive
again, the system never recovers as the connection pool is out-of-resources.
Through debugging, we believe the following causes the connection leak in
ActiveMQs handing:
{quote}Caused by: javax.jms.JMSException: java.lang.NullPointerException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1403)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1436)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.TransactionContext.rollback(TransactionContext.java:538)
~[activemq-client-5.15.11.jar:5.15.11]
... 134 more
Caused by: java.lang.NullPointerException
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.commitRemove(JDBCPersistenceAdapter.java:795)
~[activemq-jdbc-store-5.15.11.jar:5.15.11]
at
org.apache.activemq.store.jdbc.JdbcMemoryTransactionStore.rollback(JdbcMemoryTransactionStore.java:171)
~[activemq-jdbc-store-5.15.11.jar:5.15.11]
at
org.apache.activemq.transaction.XATransaction.rollback(XATransaction.java:146)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransactionBroker.rollbackTransaction(TransactionBroker.java:257)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection.processRollbackTransaction(TransportConnection.java:553)
~[activemq-broker-5.15.11.jar:5.15.11]
at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:104)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
~[activemq-broker-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
~[activemq-client-5.15.11.jar:5.15.11]
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
~[activemq-client-5.15.11.jar:5.15.11]
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
~[activemq-client-5.15.11.jar:5.15.11]
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
~[activemq-client-5.15.11.jar:5.15.11]
... 1 more
{quote}
By overloading the method 'commitRemoved(...)' in 'JDBCPersistenceAdapter' and
converting the NullPointerException above to an IOException, the connection
handling code behaves as expected, we see no connection leak, and the system
recovers correctly after the load abnormality has passed.
There is a very large number of things going wrong when these NPEs occur and
its near impossible for us (not being experts at ActiveMQ) to see what the
underlying cause for these exceptions are. However, for us, the most important
is that we recover-
> NPE under very high database load
> ---------------------------------
>
> Key: AMQ-7424
> URL: https://issues.apache.org/jira/browse/AMQ-7424
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 5.15.11
> Reporter: Terje Strand
> Assignee: Jean-Baptiste Onofré
> Priority: Major
> Fix For: 5.16.0, 5.15.12
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Under abnormally heavy database loads we get a lot of transactions timeouts
> in our application as one would expect. Our application uses XA with Postgres
> and ActiveMQ. The problem is that after the abnormality goes away the system
> does not recover.
> During these failures we get an NPE that causes ActiveMQ to lose a database
> connection and the connection is never returned to the connection pool
> (Hikari). After the abnormality is removed and the database is responsive
> again the system never recovers as the connection pool is out-of-resources.
> Through debugging, we believe the following causes the connection leak in
> ActiveMQs handing:
> {noformat}Caused by: javax.jms.JMSException: java.lang.NullPointerException
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1403)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1436)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.TransactionContext.rollback(TransactionContext.java:538)
> ~[activemq-client-5.15.11.jar:5.15.11]
> ... 134 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.commitRemove(JDBCPersistenceAdapter.java:795)
> ~[activemq-jdbc-store-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.store.jdbc.JdbcMemoryTransactionStore.rollback(JdbcMemoryTransactionStore.java:171)
> ~[activemq-jdbc-store-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.transaction.XATransaction.rollback(XATransaction.java:146)
> ~[activemq-broker-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.broker.TransactionBroker.rollbackTransaction(TransactionBroker.java:257)
> ~[activemq-broker-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
> ~[activemq-broker-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:149)
> ~[activemq-broker-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.broker.TransportConnection.processRollbackTransaction(TransportConnection.java:553)
> ~[activemq-broker-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:104)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
> ~[activemq-broker-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
> ~[activemq-broker-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
> ~[activemq-client-5.15.11.jar:5.15.11]
> at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
> ~[activemq-client-5.15.11.jar:5.15.11]
> ... 1 more{noformat}
> By overloading the method {{commitRemoved(...)}} in
> {{JDBCPersistenceAdapter}} and converting the {{NullPointerException}} above
> to an {{IOException}}, the connection handling code behaves as expected. We
> see no connection leak, and the system recovers correctly after the load
> abnormality has passed.
> There is a very large number of things going wrong when these NPEs occur and
> its near impossible for us (not being experts at ActiveMQ) to see what the
> underlying cause for these exceptions are. However, for us, the most
> important is that we recover.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
