Exception thrown during commit leads to message loss
----------------------------------------------------
Key: AMQ-2880
URL: https://issues.apache.org/activemq/browse/AMQ-2880
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.4.0
Reporter: Stan Lewis
Attachments: test-case.txt
In cases where JDBC persistence is used and the database server is under a fair
bit of load it's sometimes possible for table/row locks to time out, which
means you'll see exceptions such as:
java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting
transaction
at
com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1693)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1108)
at
org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at
org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at
org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:98)
at
org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:80)
at
org.apache.activemq.store.jdbc.TransactionContext.commit(TransactionContext.java:161)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.commitTransaction(JDBCPersistenceAdapter.java:504)
at
org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:109)
at
org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:220)
at org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:73)
at
org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:176)
at
org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
at
org.apache.activemq.broker.TransportConnection.processCommitTransactionTwoPhase(TransportConnection.java:430)
at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:102)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:309)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:217)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:201)
at java.lang.Thread.run(Thread.java:636)
In this case the connection to the database is fine and what we should do is
retry the transaction as it's a temporary failure condition. Instead what
happens is the broker moves to the next message in the queue, leaving the
current message in the database. The message won't show up in the web console
and cannot be consumed by any consumers until the broker is restarted.
Attached is a test case that simulates the error condition in a controlled
fashion by using a subclassed JDBCPersistenceAdapter that will throw an
exception in commitTransaction as necessary. So the producer sends 10 messages
and then the consumer tries to consume them, during this time the persistence
adapter will throw an exception during commitTransaction. Then the condition
is cleared and the consumer can consume all 10 messages, however the consumer
only consumes 9 messages, the 1st message in the sequence is still in the
database. Hopefully the logging makes this clear.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
