[
https://issues.apache.org/jira/browse/QPID-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201507#comment-15201507
]
Keith Wall edited comment on QPID-7154 at 3/29/16 11:12 AM:
------------------------------------------------------------
(Previous incorrect analysis deleted).
The issue is that the dead letter code path
{{QueueEntryImpl#routeToAlternative}} does not in the case the entry was
acquired by a consumer, lock the consumer's acquisition. Management operations
(such as #clearQueue) are allowed to steal consumers' acquisitions unless they
are locked. It this hole through which the problem occurred.
The fix is to have the code lock the acquisition, if it acquisition belongs to
a consumer.
This problem has existed since QPID-3978 (0.30).
was (Author: k-wall):
In 6.0.1 the issue is isolated to the JMX layer (which is disabled by default).
The {{QueueMBean#clearQueue}} implements its own queue clearing mechanism. It
uses single transaction to clear the whole queue, but omits to acquire the
queue entries. This means that two threads can be trying to act on same store
record at once. In the user's case, the queue was very deep, so the
transaction will have been very long running. The messaging thread (which had
acquired the queue entry) was also trying to dequeue it. It was blocked at
store level by the long running clearQueue, until BDB timed out the
transaction. AMQ purgeQueue and HTTP clearQueue are not affected.
In the earlier releases (0.32, 0.30, 0.22, 0.18) the same problem existed on
both JMX and HTTP paths.
> Dead lettering of a message may timeout at store level, causing unexpected
> Broker shutdown
> ------------------------------------------------------------------------------------------
>
> Key: QPID-7154
> URL: https://issues.apache.org/jira/browse/QPID-7154
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.30, 0.32, qpid-java-6.0.1
> Reporter: Keith Wall
> Fix For: qpid-java-6.0.2
>
>
> The following was reported by a user using a 0.32 derivative.
> The background was an operator was attempting to clear queue with very large
> depth using JMX management {{QueueMBean#clearQueue}} as the same time as a
> messaging client (consumer - connected to the same queue) was rejecting
> messages which were being routed to another queue and then dequeued.
> The message client (consumer) store level transaction timed out. This caused
> the Broker to fail (uncaught exception).
> {code}
> com.sleepycat.je.LockTimeoutException: (JE 5.0.104) Lock expired. Locker
> 792611890 1557851806_IoReceiver - /xxx.xx.xx.xx:32962_Txn: waited for lock on
> database=QUEUE_ENTRIES LockAddr:1959781711 LSN=0x74a70/0x94db8e type=WRITE
> grant=WAIT_NEW timeoutMillis=500 startTime=1458223769602 endTime=1458223770102
> Owners: [<LockInfo locker="1466269883 1557851804_RMI TCP
> Connection(16333)-xx.xx.xx.xx_Txn" type="WRITE"/>]
> Waiters: []
>
> at
> com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:665)
> at
> com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
> at
> com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
> at
> com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
> at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
> at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:522)
> at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
> at
> com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2642)
> at
> com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2443)
> at
> com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:2168)
> at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2822)
> at
> com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2732)
> at com.sleepycat.je.Cursor.searchNoDups(Cursor.java:2586)
> at com.sleepycat.je.Cursor.search(Cursor.java:2553)
> at
> com.sleepycat.je.Database.deleteInternal(Database.java:1151)
> at com.sleepycat.je.Database.delete(Database.java:1080)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.dequeueMessage(AbstractBDBMessageStore.java:819)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.access$1000(AbstractBDBMessageStore.java:71)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$BDBTransaction.dequeueMessage(AbstractBDBMessageStore.java:1496)
> at
> org.apache.qpid.server.txn.LocalTransaction.dequeue(LocalTransaction.java:110)
> at
> org.apache.qpid.server.queue.QueueEntryImpl.routeToAlternate(QueueEntryImpl.java:463)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.deadLetter(AMQChannel.java:1795)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.receiveBasicReject(AMQChannel.java:2431)
> at
> org.apache.qpid.framing.BasicRejectBody.process(BasicRejectBody.java:123)
> at
> org.apache.qpid.codec.ServerDecoder.processMethod(ServerDecoder.java:188)
> at
> org.apache.qpid.codec.AMQDecoder.processFrame(AMQDecoder.java:388)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.doProcessFrame(BrokerDecoder.java:113)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.access$000(BrokerDecoder.java:36)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:79)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:75)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.processFrame(BrokerDecoder.java:74)
> at
> org.apache.qpid.codec.AMQDecoder.processInput(AMQDecoder.java:370)
> at
> org.apache.qpid.codec.AMQDecoder.decodeBuffer(AMQDecoder.java:259)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine$2.run(AMQProtocolEngine.java:334)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine$2.run(AMQProtocolEngine.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine.received(AMQProtocolEngine.java:310)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine.received(AMQProtocolEngine.java:95)
> at
> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:153)
> at
> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:51)
> at
> org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161)
> at java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
