[
https://issues.apache.org/jira/browse/QPID-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714937#comment-16714937
]
Alex Rudyy commented on QPID-7918:
----------------------------------
I played with idea of listening for notifications from MemoryPoolMXBean and GC
MBean Full GC events to get the heap usage and enforce flow control if
required, but, I could not make it working reliably. The approach worked more
or less OK with java8 and CMS GC but it did not work that well with java 11 and
G1 GC. With G1 and java11, the full GC notifications are not issued reliably on
running out of heap memory. As result, the flow control was not enforced and
broker was crashing with OOM errors.
I think, that better approach would be to estimate the heap usage by queue
entries by implementing special methods on queues like {{estimateHeapUsage}}.
Thus, every queue can estimate the consumed heap size. I looked into this
approach as well but it seems it is a bit hard to get accurate estimation of
heap usage, as it is requires to traverse the object graph in memory. The
latter can be achieved either by changing field accessibility or by calling
some methods on related objects to get their heap estimation. The former
results in warnings in java11 about inappropriate use of API and the latter
requires that each QueueEntry and its ServerMessage and ServerMessageMetadata
implementations would have to provide methods for estimating of heap usage. I
attached the in-progress patch with the latter approach. The patch requires
more work but it illustrates the idea.
I am going to descope this JIRA from 7.1 and try to implement this JIRA in one
of follow up versions.
> [Broker-J] Respond to low heap memory condition by throttling producers.
> ------------------------------------------------------------------------
>
> Key: QPID-7918
> URL: https://issues.apache.org/jira/browse/QPID-7918
> Project: Qpid
> Issue Type: Improvement
> Components: Broker-J
> Reporter: Keith Wall
> Priority: Major
> Fix For: qpid-java-broker-7.1.0
>
>
> Currently, if the cumulative size of messages headers/contents cached in
> direct memory exceeds a threshold, the Java Broker responds by flowing the
> messages to disk and evicting them from memory. This guards the condition of
> excessive *message payload* causing an out-of-memory situation.
> However the Broker also maintains a linked list representing every queues in
> the JVMs heap memory. Each message is represented by a small node in the
> linked list. The Broker's design currently means that the entire linked list
> must reside in (heap) memory (even if the message's content has been paged
> out by the flow to disk mechanism). This gives rise to the the possibility
> that the linked lists backing the queue could exhaust all heap and the Broker
> could fail with an OOM (heap) condition.
> The long term aim is to allow sections linked list to be paged out to
> eliminate this problem, but this would necessitate a deep redesign.
> As a interim measure, it appears that the the JVM's Memory bean
> (MemoryPoolMXBean) could be used to notify the Broker when after a garbage
> collection cycle heap memory usage is above a threshold. The Broker could
> respond to this event by throttling producer, thus allow consumes to catch up
> and hopefully avoiding the failure. The Broker would need to relinquish flow
> control once memory usage had fallen again.
> Consideration would need to be given for different JVMs and different GC
> implementations. Consideration must also be given for use cases where the
> Broker is embedded.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
