Hello all.
I know it’s not ideal but the broker is doing just fine (except for the leak
issue of course).
I’ve tried upgrading to 2.30.0 and the broker still ends up on its knees given
enough load and only a little heap. In my testing case I’ve limited the heap
size to 64 MiB so that I wouldn't have to wait for days for things to happen,
and also the consumer creation/disposal rate is different to the production
state. Here’s a very simple code which manages to take down the 64 MiB broker
in about 10 to 15 minutes on Java 11 and recent Windows 10:
final String queueName = "clouedi-kestra";
final String filter = "";
final Thread[] threads = new Thread[16];
for (int i = 0; i < threads.length; i++) {
threads[i] = new Thread(() -> {
try (
ServerLocator locator =
ActiveMQClient.createServerLocator("tcp://localhost:61616");
ClientSessionFactory sf = locator.createSessionFactory();
ClientSession session = sf.createSession(false, true);
) {
while (!session.isClosed()) {
try (ClientConsumer consumer =
session.createConsumer(SimpleString.toSimpleString(queueName),
SimpleString.toSimpleString(filter), 0, 0, false)) {
consumer.receive(1);
}
}
} catch (Exception e) {
throw new RuntimeException(e);
}
});
threads[i].start();
}
for (Thread thread : threads) {
thread.join();
}
Big thanks for your help!
Jan
From: Arthur Naseef<mailto:a...@amlinv.com>
Sent: čtvrtek 10. srpna 2023 21:11
To: users@activemq.apache.org<mailto:users@activemq.apache.org>
Subject: Re: Hunting memory leaks
Creating a consumer only to consume 1 message is not ideal - there's a lot
of overhead and work on the broker side when consumers are created.
With that said, since the consumer should be getting closed properly, that
should not cause a leak.
So first, I would prioritize the version update. Second, I would consider
changing the use of consumers so they are longer-lived - preferrably only
being removed once the application needs to stop consuming.
If there is a need to throttle and/or control threading and parallel
processing of messages, perhaps Camel would be a good fit.
Hope this helps.
Art
On Wed, Aug 9, 2023 at 10:44 PM Jan Šmucr <jan.sm...@aimtecglobal.com>
wrote:
> Hello all. Thank you for your insights.
>
>
> * I’m using the core Java library.
>
>
> * Consumers are being created once per poll but reused if there are
> multiple inbound files to deal with. I create consumers like
>
> try (final consumer = createConsumer(session, params)) {
>
> // ...
> }
>
> so I expect them to be closed automatically.
>
>
> * I don’t use JMS, but the core sessions are used one per thread. The
> number of sessions opened and reported by Artemis doesn’t change over time.
>
>
> * I cannot reproduce the issue yet. It’s a production cluster, so
> today I’m going to set up my own playground.
>
>
> Jan
>
> From: Justin Bertram<mailto:jbert...@apache.org>
> Sent: středa 9. srpna 2023 17:41
> To: users@activemq.apache.org<mailto:users@activemq.apache.org>
> Subject: Re: Hunting memory leaks
>
> I echo Tim's recommendation to use the latest release, but I don't mean to
> say that will certainly resolve the problem.
>
> I can't say if you're doing anything wrong without more information. Can
> you answer the following questions?
>
> - What client library are you using?
> - How often are consumers being created?
> - Are consumers being closed properly once they are no longer needed?
> - Are JMS sessions being used concurrently from multiple threads?
> - Do you have a way to reproduce this that you can provide to me? A
> reproducer would make diagnosing this issue much simpler.
>
> Entries to the list of filter strings are added when a consumer is created
> and removed when a consumer is closed so at first glance it appears you're
> leaking consumers.
>
>
> Justin
>
> On Wed, Aug 9, 2023 at 7:07 AM Jan Šmucr <jan.sm...@aimtecglobal.com>
> wrote:
>
> > Hello.
> > I’m using a simple master-slave Artemis 2.26.0 cluster, and I’m noticing
> > heap usage growing more and more each day no matter the throughput.
> There’s
> > about 670 sessions at the same time opened for producers and consumers.
> > Consumers are polling queues on regular basis, some once a second
> (meaning
> > 1s timeout), some less often. This is by design and cannot be altered.
> All
> > client resources are being reused as much as possible. Usually there’s a
> > thread pool and the threads have a session opened, and wait for tasks to
> be
> > available to them.
> > It appears to me that the more consumers there is the faster the server
> > heap depletes.
> > Now, I’m not very familiar with leak hunting apps, so all I have are tiny
> > hints that it may have something to do with filter strings not being
> reused
> > and/or thrown away when not needed any more. I don’t know if I can post a
> > screenshot here so I uploaded it here: https://snipboard.io/LHifUK.jpg
> > This is from a heap dump opened in JMC JOverflow plugin.
> > Is there something obvious that I’m doing wrong? Do you have any clues on
> > what is going on here?
> > Thank you.
> > Jan.
> >
> >
>
>
