[
https://issues.apache.org/jira/browse/QPID-7977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keith Wall updated QPID-7977:
-----------------------------
Attachment: MultiNodeTest.tar.bz2
> [Java Broker] [HA] Direct memory leak possible when mastership is lost
> ----------------------------------------------------------------------
>
> Key: QPID-7977
> URL: https://issues.apache.org/jira/browse/QPID-7977
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: qpid-java-6.0, qpid-java-6.1, qpid-java-broker-7.0.0
> Reporter: Keith Wall
> Priority: Minor
> Attachments: MultiNodeTest.tar.bz2
>
>
> During the work of QPID-7832, it was noticed that there is a potential for
> the direct memory used for uncommitted messages to be leaked when a node
> loses the mastership.
> The test that exposed the direct leak is:
> {{org.apache.qpid.server.store.berkeleydb.replication.MultiNodeTest#testLossOfMasterNodeCausesClientToFailover}}
> I think the problem is this. When mastership is lost, the underlying
> database operations that occur after fail. One possible place this can occur
> is the is the rolling back open transactions. The rollback causes the store
> to try and remove the messages
> {{org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.StoredBDBMessage#remove}}.
> The underlying database operation fails on line
> {{org/apache/qpid/server/store/berkeleydb/AbstractBDBMessageStore.java:1085}}.
> This prevents the direct memory associated with the messages from being
> released. One possible fix is a simple try/finally. There might be other
> places that need similar attention.
> The operation of the group is unaffected, and there is no message loss. The
> JVM should eventually release the direct memory itself.
> The non-HA use-case is unaffected.
> Example database exception{noformat}
> 2017-10-18 17:03:32,544 BROKER-52 ERROR [IO-/127.0.0.1:62547]
> o.a.q.s.s.b.BDBConfigurationStore Unexpected BDB exception
> com.sleepycat.je.rep.ReplicaWriteException: (JE 7.4.5) Problem closing
> transaction 14. The current state is:UNKNOWN. The node transitioned to this
> state at:Wed Oct 18 17:03:32 BST 2017
> at
> com.sleepycat.je.rep.txn.ReadonlyTxn.disallowReplicaWrite(ReadonlyTxn.java:114)
> at
> com.sleepycat.je.rep.txn.ReadonlyTxn.lockInternal(ReadonlyTxn.java:86)
> at com.sleepycat.je.txn.Locker.nonBlockingLock(Locker.java:534)
> at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3606)
> at
> com.sleepycat.je.dbi.CursorImpl.deleteCurrentRecord(CursorImpl.java:900)
> at com.sleepycat.je.Cursor.deleteNoNotify(Cursor.java:2412)
> at com.sleepycat.je.Cursor.deleteInternal(Cursor.java:2353)
> at com.sleepycat.je.Database.deleteInternal(Database.java:1169)
> at com.sleepycat.je.Database.delete(Database.java:1128)
> at com.sleepycat.je.Database.delete(Database.java:1230)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:273)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.remove(AbstractBDBMessageStore.java:1085)
> at
> org.apache.qpid.server.message.AbstractServerMessageImpl.decrementReference(AbstractServerMessageImpl.java:118)
> at
> org.apache.qpid.server.message.AbstractServerMessageImpl.access$500(AbstractServerMessageImpl.java:37)
> at
> org.apache.qpid.server.message.AbstractServerMessageImpl$Reference.release(AbstractServerMessageImpl.java:309)
> at
> org.apache.qpid.server.message.RoutingResult$1.onRollback(RoutingResult.java:131)
> at
> org.apache.qpid.server.txn.LocalTransaction$4.onRollback(LocalTransaction.java:315)
> at
> org.apache.qpid.server.txn.LocalTransaction.doRollbackActions(LocalTransaction.java:386)
> at
> org.apache.qpid.server.txn.LocalTransaction.rollback(LocalTransaction.java:462)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.close(AMQChannel.java:802)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.close(AMQChannel.java:778)
> at
> org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8Impl.closeAllChannels(AMQPConnection_0_8Impl.java:487)
> at
> org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8Impl.completeAndCloseAllChannels(AMQPConnection_0_8Impl.java:521)
> at
> org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8Impl.closed(AMQPConnection_0_8Impl.java:668)
> at
> org.apache.qpid.server.transport.MultiVersionProtocolEngine.closed(MultiVersionProtocolEngine.java:101)
> at
> org.apache.qpid.server.transport.NonBlockingConnection.shutdown(NonBlockingConnection.java:385)
> at
> org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:313)
> at
> org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:134)
> at
> org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:563)
> at
> org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:354)
> at
> org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:97)
> at
> org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:521)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.qpid.server.bytebuffer.QpidByteBuffer.lambda$null$0(QpidByteBuffer.java:1388)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
