Marco Geri created QPID-8789:
--------------------------------
Summary: [Broker-J] Pooled buffers leaked after a BDB flow to disk
make the broker flow every message to disk until it is restarted
Key: QPID-8789
URL: https://issues.apache.org/jira/browse/QPID-8789
Project: Qpid
Issue Type: Bug
Components: Broker-J
Affects Versions: qpid-java-broker-10.1.1, qpid-java-broker-10.1.0
Environment: Broker-J 10.1.0, Java 21, BDB virtual host, AMQP 1.0
clients. The affected code is unchanged on main.
Reporter: Marco Geri
Attachments: QPID-XXXX-proposed-fix.patch,
QPID-XXXX-reproducer-test.patch
h2. Problem
With a BDB virtual host and AMQP 1.0 messages, every flow to disk can leak
pooled direct buffers from the accounting. The JVM frees their memory, but
{{numberOfBuffersInUse}} never goes down, and
{{getAllocatedDirectMemorySize()}} is derived from it. Once that value passes
{{broker.flowToDiskThreshold}}, the broker flows every incoming message to disk
and reloads it from the store on delivery. Every such delivery flows the
message to disk again and leaks more, so the broker never recovers: it stays
very slow until it is restarted.
In production (16 GiB max direct memory, messages routed to several queues,
browsers, message filters) the counter reached 202,512 buffers (49 GiB) while
the JVM had 1.9 GiB of direct memory reserved. The drift started after a mass
flow to disk; on days without one, the two values match.
h2. Reproduce
{{QPID-XXXX-reproducer-test.patch}} adds {{BDBFlowToDiskMetaDataTest}}
({{broker-plugins/amqp-1-0-bdb-store}}), on a real {{BDBMessageStore}}: take
the metadata of a message, flow the message to disk, copy a section as a
delivery does, remove the message. On {{main}} it fails:
{noformat}
Pooled buffers still counted as in use after the message was removed ==>
expected: <0> but was: <1>
{noformat}
It passes if the section is not read after the flow to disk.
h2. Cause
{{StoredBDBMessage#flowToDisk()}} calls {{MessageDataRef#clear(false)}}, which
releases the encoded form of the metadata and drops the reference:
{code:java}
_metaData.clearEncodedForm();
...
_metaData = null;
{code}
Threads that obtained the metadata earlier through
{{StoredBDBMessage#getMetaData()}} keep using it: consumers of other queues
delivering the same message, browsers, filters. When one of them reads a
section, {{AbstractSection#encodeIfNecessary()}} re-encodes it into a new
pooled buffer. This happens in {{copy()}} on every delivery, in
{{getEncodedForm()}}, {{writeTo()}} and {{getValue()}}. Nothing owns that
metadata any more, so the buffer is never disposed.
Flow to disk on BDB happens in {{AbstractQueue#enqueue}} above the threshold,
and in {{AbstractServerMessageImpl#getContent}} on every delivery of a message
whose content was on disk. That is why the leak feeds itself.
h2. Attachments
* *{{QPID-XXXX-reproducer-test.patch}}*: the test above. It fails on {{main}}.
* *{{QPID-XXXX-proposed-fix.patch}}*: a first attempt at a fix, meant only as
a starting point. Once {{clearEncodedForm()}} has been called,
{{AbstractSection}} re-encodes into a heap buffer, which needs no dispose.
Sections that were never cleared are unchanged. With it, the test passes, and
so do the existing tests of {{amqp-1-0-protocol}}, {{bdbstore}} and
{{amqp-1-0-bdb-store}} (including with {{-Dprofile=java-bdb.1-0}}).
You know this code far better than we do, so please treat the patch as a
suggestion to be reassessed rather than a finished fix. In particular, we are
not sure whether the right place for it is the section or the store. Another
option would be for {{MessageDataRef#clear(false)}} to keep owning the metadata
until the message is removed, at the cost of keeping decoded metadata on the
heap for messages on disk. There may well be a better approach we have not
considered. We are happy to test any alternative against our workload, and
thank you for taking a look.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
