Alex Rudyy created QPID-8387:
--------------------------------
Summary: [Broker-J] JDBC based message stores do not wait for
asynchronous message removal threads to complete when store is being closed
Key: QPID-8387
URL: https://issues.apache.org/jira/browse/QPID-8387
Project: Qpid
Issue Type: Bug
Components: Broker-J
Affects Versions: qpid-java-broker-7.1.4, qpid-java-broker-7.1.3,
qpid-java-broker-7.1.2, qpid-java-broker-7.1.1, qpid-java-broker-7.1.0,
qpid-java-broker-7.1.5, qpid-java-broker-7.1.6
Reporter: Alex Rudyy
The asynchronous message removal was added into JDBC based message stores
(Derby and JDBC) in version 7.1.0 of Qpid Broker-J as part of
[QPID-8242|https://issues.apache.org/jira/browse/QPID-8242]. The message
removal now happens asynchronously in internal threads of JDBC based message
stores.
However, when store is closed, the closing functionality does not wait for the
asynchronous message removal threads to complete. This issue causes several
side effects:
* the messages scheduled for removal might not be removed due to the close of
the VirtualHost. Though, it is not an issue, as Broker on restart deletes all
orphaned messages
* the asynchronous removal might result in ServerScopedRuntimeException being
thrown from the removal thread. At the moment, those
ServerScopedRuntimeException are ignored, but the intention of
ServerScopedRuntimeException is to halt the JVM, and, this does not happen
* The restart of VirtualHost in integration tests can hang due to issues with
store recovery
The following exception can be thrown from the tests in
{{PersistentMessagingTest}} when any of derby profiles is used
{noformat}
019-11-29 12:33:23,062 DEBUG [Broker-Config] o.a.q.s.m.AbstractConfiguredObject
All children closed DerbyVirtualHostNodeImplWithAccessChecking :
PersistentMessagingTest_transactedAcknowledgementPersistence
2019-11-29 12:33:23,070 ERROR
[PersistentMessagingTest_transactedAcknowledgementPersistence-store-1]
o.a.q.s.u.ServerScopedRuntimeException Error removing messages with ids [3]
from database: Java exception: ':
org.apache.derby.shared.common.error.ShutdownException'.
java.sql.SQLException: Java exception: ':
org.apache.derby.shared.common.error.ShutdownException'.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.close(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.close(Unknown Source)
at
org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore.removeMessages(AbstractJDBCMessageStore.java:505)
at
org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore.lambda$scheduleMessageRemoval$0(AbstractJDBCMessageStore.java:461)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.derby.iapi.error.StandardException: Java exception: ':
org.apache.derby.shared.common.error.ShutdownException'.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 18 common frames omitted
Caused by: org.apache.derby.shared.common.error.ShutdownException:
at
org.apache.derby.iapi.services.context.ContextManager.checkInterrupt(Unknown
Source)
at
org.apache.derby.iapi.services.context.ContextManager.getContext(Unknown Source)
at
org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.getStatementContext(Unknown
Source)
at
org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.doRollback(Unknown
Source)
at
org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.userRollback(Unknown
Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.rollback(Unknown
Source)
... 11 common frames omitted
2019-11-29 12:33:24,673 INFO [Broker-Config] q.m.c.close <<UNKNOWN>>
[vh(/PersistentMessagingTest_transactedAcknowledgementPersistence)/ms(DerbyConfigurationStore)]
CFG-1003 : Closed
2019-11-29 12:33:24,673 INFO [main] o.a.q.t.u.EmbeddedBrokerPerClassAdminImpl
Starting VirtualHostNode for restart
{noformat}
As minimum, the following changes need to be implemented
* {{ServerScopedRuntimeException}} thrown from async message removal threads
should halt Broker JVM
* The message store close functionality should either wait for in-progress
removal to complete, or, cancel removal tasks gracefully
* The issue with test hanging should be resolved
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
