I highly recommend you using check-leak.. you would have found what's leaking already.
https://github.com/check-leak/check-leak java -jar check-leak-0.10.jar remote --pid <PID> --report <reportoutput> --sleep 5000 ( I suggest using 5 seconds for your test) I would even write a unit-test for memory-leaks. On Fri, Aug 11, 2023 at 10:06 AM Jan Šmucr <jan.sm...@aimtecglobal.com> wrote: > > So I’m getting a bit closer. The leak is in PostOfficeImpl and QueueInfo. > QueueInfo contains the filterStrings List which appears to contain a list of > filters used by consumers subscribed to that queue. However, for some reason > this list is updated in a very strange way. For one or two consumers there > are no CONSUMER_CREATED and CONSUMER_CLOSED core notifications which > PostOfficeImpl would receive and update the list accordingly (also managing > the list from outside of QueueInfo is quite weird). > From the 3rd consumer the management messages start flowing, and here comes > the catch: The CONSUMER_CREATED message contains _AMQ_FilterString = "" > whereas the CONSUMER_CLOSED message contains AMQ_FilterString = null. So the > filterStrings List keeps filling up by empty strings because these don’t get > removed based on a null value. > > Jan > > From: Jan Šmucr<mailto:jan.sm...@aimtecglobal.com> > Sent: pátek 11. srpna 2023 9:43 > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > Subject: RE: Hunting memory leaks > > Hello all. > > I know it’s not ideal but the broker is doing just fine (except for the leak > issue of course). > > I’ve tried upgrading to 2.30.0 and the broker still ends up on its knees > given enough load and only a little heap. In my testing case I’ve limited the > heap size to 64 MiB so that I wouldn't have to wait for days for things to > happen, and also the consumer creation/disposal rate is different to the > production state. Here’s a very simple code which manages to take down the 64 > MiB broker in about 10 to 15 minutes on Java 11 and recent Windows 10: > > final String queueName = "clouedi-kestra"; > final String filter = ""; > final Thread[] threads = new Thread[16]; > for (int i = 0; i < threads.length; i++) { > threads[i] = new Thread(() -> { > try ( > ServerLocator locator = > ActiveMQClient.createServerLocator("tcp://localhost:61616"); > ClientSessionFactory sf = locator.createSessionFactory(); > ClientSession session = sf.createSession(false, true); > ) { > while (!session.isClosed()) { > try (ClientConsumer consumer = > session.createConsumer(SimpleString.toSimpleString(queueName), > SimpleString.toSimpleString(filter), 0, 0, false)) { > consumer.receive(1); > } > } > } catch (Exception e) { > throw new RuntimeException(e); > } > }); > threads[i].start(); > } > for (Thread thread : threads) { > thread.join(); > } > > Big thanks for your help! > Jan > > From: Arthur Naseef<mailto:a...@amlinv.com> > Sent: čtvrtek 10. srpna 2023 21:11 > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > Subject: Re: Hunting memory leaks > > Creating a consumer only to consume 1 message is not ideal - there's a lot > of overhead and work on the broker side when consumers are created. > > With that said, since the consumer should be getting closed properly, that > should not cause a leak. > > So first, I would prioritize the version update. Second, I would consider > changing the use of consumers so they are longer-lived - preferrably only > being removed once the application needs to stop consuming. > > If there is a need to throttle and/or control threading and parallel > processing of messages, perhaps Camel would be a good fit. > > Hope this helps. > > Art > > > On Wed, Aug 9, 2023 at 10:44 PM Jan Šmucr <jan.sm...@aimtecglobal.com> > wrote: > > > Hello all. Thank you for your insights. > > > > > > * I’m using the core Java library. > > > > > > * Consumers are being created once per poll but reused if there are > > multiple inbound files to deal with. I create consumers like > > > > try (final consumer = createConsumer(session, params)) { > > > > // ... > > } > > > > so I expect them to be closed automatically. > > > > > > * I don’t use JMS, but the core sessions are used one per thread. The > > number of sessions opened and reported by Artemis doesn’t change over time. > > > > > > * I cannot reproduce the issue yet. It’s a production cluster, so > > today I’m going to set up my own playground. > > > > > > Jan > > > > From: Justin Bertram<mailto:jbert...@apache.org> > > Sent: středa 9. srpna 2023 17:41 > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > > Subject: Re: Hunting memory leaks > > > > I echo Tim's recommendation to use the latest release, but I don't mean to > > say that will certainly resolve the problem. > > > > I can't say if you're doing anything wrong without more information. Can > > you answer the following questions? > > > > - What client library are you using? > > - How often are consumers being created? > > - Are consumers being closed properly once they are no longer needed? > > - Are JMS sessions being used concurrently from multiple threads? > > - Do you have a way to reproduce this that you can provide to me? A > > reproducer would make diagnosing this issue much simpler. > > > > Entries to the list of filter strings are added when a consumer is created > > and removed when a consumer is closed so at first glance it appears you're > > leaking consumers. > > > > > > Justin > > > > On Wed, Aug 9, 2023 at 7:07 AM Jan Šmucr <jan.sm...@aimtecglobal.com> > > wrote: > > > > > Hello. > > > I’m using a simple master-slave Artemis 2.26.0 cluster, and I’m noticing > > > heap usage growing more and more each day no matter the throughput. > > There’s > > > about 670 sessions at the same time opened for producers and consumers. > > > Consumers are polling queues on regular basis, some once a second > > (meaning > > > 1s timeout), some less often. This is by design and cannot be altered. > > All > > > client resources are being reused as much as possible. Usually there’s a > > > thread pool and the threads have a session opened, and wait for tasks to > > be > > > available to them. > > > It appears to me that the more consumers there is the faster the server > > > heap depletes. > > > Now, I’m not very familiar with leak hunting apps, so all I have are tiny > > > hints that it may have something to do with filter strings not being > > reused > > > and/or thrown away when not needed any more. I don’t know if I can post a > > > screenshot here so I uploaded it here: https://snipboard.io/LHifUK.jpg > > > This is from a heap dump opened in JMC JOverflow plugin. > > > Is there something obvious that I’m doing wrong? Do you have any clues on > > > what is going on here? > > > Thank you. > > > Jan. > > > > > > > > > > > -- Clebert Suconic
