Thanks for your reply. The metrics used for this investigation is collected by datadog against the broker JMX endpoint, and the kubernetes metrics-server.
"container" bytes are what I call the metrics collected by the metrics-server, so what kubernetes refer to as used container memory. It is larger than JVM heap, and a large portion is AnonPages when checking /proc/meminfo. Heap dumps are generally quite small, in the order of hundreds of MB, but have large (GBs) "unreachable" memory space (according to Eclipse Memory Analyzer) which I guess is memory waiting for garbage collection. The JVM runtime metrics in the artemis console shows a heap usage that gradually grows from ~1GB with a constant load up to ~4GB, and then get (what is assume) garbage collected once per hour back down to ~1GB. This is when very low address space is usage. The numbers I find when looking at the JVM heap are hard to track and are not entirely consistent since our issues only happen during high load and I can't do the heap dumps on our live environment and have to try to simulate it. What I refer to as address size is the memory allocated above this base line 1GB heap memory, which I understand is used to store messages being processed. The only thing that sticks out when doing this investigation into our memory issues is that "overhead" of ~100 container bytes per address size byte. I understand that messages have overhead, but find that 100 times is not resonable and that I should rather be a fixed amount per message. I don't know if I can add images here, but I'll try. At 1 the container was started. [cid:28ca1772-c01f-4015-97fa-93a045bec223][cid:3f16941b-9977-46c8-951a-22ca03918a68][cid:9cecc967-9a07-4115-ab39-ce7bc0df71b9] ________________________________ From: Justin Bertram <jbert...@apache.org> Sent: Friday, September 8, 2023 15:16 To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: Very high address space container memory usage [You don't often get email from jbert...@apache.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] > ...each "address size" byte takes 40- 160 "container" bytes. Can you elaborate on how this was calculated? Also, what exactly do you mean by "'container' bytes"? Are you talking about JVM heap? Keep in mind that as far as the broker is concerned memory usage won't change when running in a container or on bare metal. It's also worth noting that the messages represented by the address size are far from the only thing in memory. Every object dedicated to running the broker is also in memory as well. Have you tried acquiring any heap dumps? If so, did you analyze them? If so, what was the result of that analysis? Justin On Fri, Sep 8, 2023 at 2:06 AM Tobias Månsson (external) <tobias.mans...@husqvarnagroup.com.invalid> wrote: > We have a Artemis 2.30.0 2 node cluster setup running in amazoncorretto:17 > container image (Java 17 JRE), with persistance disabled. > > > We've had issues with high memory usage for our Artemis broker for a long > time, but recently several events raised a possible candidate to the cause. > There were sudden drops in address size correlating to container memory > drops, and calculations show that each "address size" byte takes 40- 160 > "container" bytes. > > > Is this normal and intended? > > > Should the parameter max-size-bytes be multiplied by say 100 to get > "actually" container memory usage? > > > The information in this email may be confidential and/or legally > privileged. It has been sent for the sole use of the intended recipient(s). > If you are not an intended recipient, you are strictly prohibited from > reading, disclosing, distributing, copying or using this email or any of > its contents, in any way whatsoever. If you have received this email in > error, please contact the sender by reply email and destroy all copies of > the original message. Please also be advised that emails are not a secure > form for communication, and may contain errors. > The information in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not an intended recipient, you are strictly prohibited from reading, disclosing, distributing, copying or using this email or any of its contents, in any way whatsoever. If you have received this email in error, please contact the sender by reply email and destroy all copies of the original message. Please also be advised that emails are not a secure form for communication, and may contain errors.
