[
https://issues.apache.org/jira/browse/QPID-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniil Kirilyuk reassigned QPID-8789:
-------------------------------------
Assignee: Daniil Kirilyuk
> [Broker-J] Pooled buffers leaked after a BDB flow to disk make the broker
> flow every message to disk until it is restarted
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-8789
> URL: https://issues.apache.org/jira/browse/QPID-8789
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-broker-10.1.0, qpid-java-broker-10.1.1
> Environment: Broker-J 10.1.0, Java 21, BDB virtual host, AMQP 1.0
> clients. The affected code is unchanged on main.
> Reporter: Marco Geri
> Assignee: Daniil Kirilyuk
> Priority: Major
> Attachments: QPID-XXXX-proposed-fix.patch,
> QPID-XXXX-reproducer-test.patch
>
>
> h2. Problem
> With a BDB virtual host and AMQP 1.0 messages, every flow to disk can leak
> pooled direct buffers from the accounting. The JVM frees their memory, but
> {{numberOfBuffersInUse}} never goes down, and
> {{getAllocatedDirectMemorySize()}} is derived from it. Once that value passes
> {{broker.flowToDiskThreshold}}, the broker flows every incoming message to
> disk and reloads it from the store on delivery. Every such delivery flows the
> message to disk again and leaks more, so the broker never recovers: it stays
> very slow until it is restarted.
> In production (16 GiB max direct memory, messages routed to several queues,
> browsers, message filters) the counter reached 202,512 buffers (49 GiB) while
> the JVM had 1.9 GiB of direct memory reserved. The drift started after a mass
> flow to disk; on days without one, the two values match.
> h2. Reproduce
> {{QPID-XXXX-reproducer-test.patch}} adds {{BDBFlowToDiskMetaDataTest}}
> ({{broker-plugins/amqp-1-0-bdb-store}}), on a real {{BDBMessageStore}}: take
> the metadata of a message, flow the message to disk, copy a section as a
> delivery does, remove the message. On {{main}} it fails:
> {noformat}
> Pooled buffers still counted as in use after the message was removed ==>
> expected: <0> but was: <1>
> {noformat}
> It passes if the section is not read after the flow to disk.
> h2. Cause
> {{StoredBDBMessage#flowToDisk()}} calls {{MessageDataRef#clear(false)}},
> which releases the encoded form of the metadata and drops the reference:
> {code:java}
> _metaData.clearEncodedForm();
> ...
> _metaData = null;
> {code}
> Threads that obtained the metadata earlier through
> {{StoredBDBMessage#getMetaData()}} keep using it: consumers of other queues
> delivering the same message, browsers, filters. When one of them reads a
> section, {{AbstractSection#encodeIfNecessary()}} re-encodes it into a new
> pooled buffer. This happens in {{copy()}} on every delivery, in
> {{getEncodedForm()}}, {{writeTo()}} and {{getValue()}}. Nothing owns that
> metadata any more, so the buffer is never disposed.
> Flow to disk on BDB happens in {{AbstractQueue#enqueue}} above the threshold,
> and in {{AbstractServerMessageImpl#getContent}} on every delivery of a
> message whose content was on disk. That is why the leak feeds itself.
> h2. Attachments
> * *{{QPID-XXXX-reproducer-test.patch}}*: the test above. It fails on
> {{main}}.
> * *{{QPID-XXXX-proposed-fix.patch}}*: a first attempt at a fix, meant only
> as a starting point. Once {{clearEncodedForm()}} has been called,
> {{AbstractSection}} re-encodes into a heap buffer, which needs no dispose.
> Sections that were never cleared are unchanged. With it, the test passes, and
> so do the existing tests of {{amqp-1-0-protocol}}, {{bdbstore}} and
> {{amqp-1-0-bdb-store}} (including with {{-Dprofile=java-bdb.1-0}}).
> You know this code far better than we do, so please treat the patch as a
> suggestion to be reassessed rather than a finished fix. In particular, we are
> not sure whether the right place for it is the section or the store. Another
> option would be for {{MessageDataRef#clear(false)}} to keep owning the
> metadata until the message is removed, at the cost of keeping decoded
> metadata on the heap for messages on disk. There may well be a better
> approach we have not considered. We are happy to test any alternative against
> our workload, and thank you for taking a look.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
