[
https://issues.apache.org/jira/browse/QPID-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201507#comment-15201507
]
Keith Wall commented on QPID-7154:
----------------------------------
In 6.0.1 the issue is isolated to the JMX layer (which is disabled by default).
The {{QueueMBean#clearQueue}} implements its own queue clearing mechanism. It
uses single transaction to clear the whole queue, but omits to acquire the
queue entries. This means that two threads can be trying to act on same store
record at once. In the user's case, the queue was very deep, so the
transaction will have been very long running. The messaging thread (which had
acquired the queue entry) was also trying to dequeue it. It was blocked at
store level by the long running clearQueue, until BDB timed out the transaction.
> JMX QueueMBean#clearQueue operation does not acquire messages prior to
> dequeue which allows for store level lock timeout
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-7154
> URL: https://issues.apache.org/jira/browse/QPID-7154
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.18, 0.22, 0.30, 0.32, qpid-java-6.0.1
> Reporter: Keith Wall
> Fix For: qpid-java-6.0.2
>
>
> The following was reported by a user using a 0.32 derivative.
> The background was an operator was attempting to clear queue with very large
> depth using JMX management (QueueMBean#clearQueue) as the same time as a
> messaging client (consumer) was rejecting messages which were being routed to
> another queue and then dequeued.
> The message client (consumer) store level transaction timed out. This caused
> the Broker to fail.
> {code}
> com.sleepycat.je.LockTimeoutException: (JE 5.0.104) Lock expired. Locker
> 792611890 1557851806_IoReceiver - /xxx.xx.xx.xx:32962_Txn: waited for lock on
> database=QUEUE_ENTRIES LockAddr:1959781711 LSN=0x74a70/0x94db8e type=WRITE
> grant=WAIT_NEW timeoutMillis=500 startTime=1458223769602 endTime=1458223770102
> Owners: [<LockInfo locker="1466269883 1557851804_RMI TCP
> Connection(16333)-xx.xx.xx.xx_Txn" type="WRITE"/>]
> Waiters: []
>
> at
> com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:665)
> at
> com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
> at
> com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
> at
> com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
> at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
> at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:522)
> at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
> at
> com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2642)
> at
> com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2443)
> at
> com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:2168)
> at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2822)
> at
> com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2732)
> at com.sleepycat.je.Cursor.searchNoDups(Cursor.java:2586)
> at com.sleepycat.je.Cursor.search(Cursor.java:2553)
> at
> com.sleepycat.je.Database.deleteInternal(Database.java:1151)
> at com.sleepycat.je.Database.delete(Database.java:1080)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.dequeueMessage(AbstractBDBMessageStore.java:819)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.access$1000(AbstractBDBMessageStore.java:71)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$BDBTransaction.dequeueMessage(AbstractBDBMessageStore.java:1496)
> at
> org.apache.qpid.server.txn.LocalTransaction.dequeue(LocalTransaction.java:110)
> at
> org.apache.qpid.server.queue.QueueEntryImpl.routeToAlternate(QueueEntryImpl.java:463)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.deadLetter(AMQChannel.java:1795)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.receiveBasicReject(AMQChannel.java:2431)
> at
> org.apache.qpid.framing.BasicRejectBody.process(BasicRejectBody.java:123)
> at
> org.apache.qpid.codec.ServerDecoder.processMethod(ServerDecoder.java:188)
> at
> org.apache.qpid.codec.AMQDecoder.processFrame(AMQDecoder.java:388)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.doProcessFrame(BrokerDecoder.java:113)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.access$000(BrokerDecoder.java:36)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:79)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:75)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.processFrame(BrokerDecoder.java:74)
> at
> org.apache.qpid.codec.AMQDecoder.processInput(AMQDecoder.java:370)
> at
> org.apache.qpid.codec.AMQDecoder.decodeBuffer(AMQDecoder.java:259)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine$2.run(AMQProtocolEngine.java:334)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine$2.run(AMQProtocolEngine.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine.received(AMQProtocolEngine.java:310)
> at
> org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine.received(AMQProtocolEngine.java:95)
> at
> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:153)
> at
> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:51)
> at
> org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161)
> at java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
