This only happened because you used the Core API directly, and passed a "" as the filter String.
In the JMS layer, there's a check to replace "" by null and that would leave that out. there's a check to add and remove existing filters from the selectors.. but in your case this is creating a leak of "" strings. You should instead pass in null instead of "" for now. We could add a layer of protection on the client and on the server replacing "" by null both client and server. As for a test... We could add a test under ./tests/leak-tests... counting the number of SimpleStrings with CheckLeak (look at other tests) and then making sure it won't go beyond a certain limit. (Do you want to send the PR yourself?) I'm not going to rush for a fix now as you can just use null instead of "" on your filter string and this leak won't happen (I already tested it). let me know if you want to send the PR.. as I don't want to duplicate your efforts. If you're not fixing it just let me know and I will do it on monday. Thanks for this ! On Sat, Aug 12, 2023 at 10:53 AM Jan Šmucr <jan.sm...@aimtecglobal.com> wrote: > > I'll leave it up to you. If you're busy, I'll have created a PR by Monday > too. 🙂 And as a bonus, I'll get better acquaintanted with the Artemis code. > > Jan > > ________________________________ > Od: Clebert Suconic <clebert.suco...@gmail.com> > Odesláno: sobota, srpna 12, 2023 4:48:08 odp. > Komu: users@activemq.apache.org <users@activemq.apache.org> > Předmět: Re: Hunting memory leaks > > The leak is because you are creating a consumer within the same > session over and over: > > https://github.com/apache/activemq-artemis/blob/064018a3e9a1ba39ddbee0bbddfed3e7fccab89c/artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/PostOfficeImpl.java#L410-L420 > > > Do you want to raise the JIRA for this? I should have a fix by monday. > > > If you keep your consumer open instead of open / close it all the > time this won't happen. But I should have a fix by monday. > > On Fri, Aug 11, 2023 at 12:24 PM Clebert Suconic > <clebert.suco...@gmail.com> wrote: > > > > I highly recommend you using check-leak.. you would have found what's > > leaking already. > > > > https://github.com/check-leak/check-leak > > > > java -jar check-leak-0.10.jar remote --pid <PID> --report > > <reportoutput> --sleep 5000 > > > > ( I suggest using 5 seconds for your test) > > > > > > > > I would even write a unit-test for memory-leaks. > > > > On Fri, Aug 11, 2023 at 10:06 AM Jan Šmucr <jan.sm...@aimtecglobal.com> > > wrote: > > > > > > So I’m getting a bit closer. The leak is in PostOfficeImpl and QueueInfo. > > > QueueInfo contains the filterStrings List which appears to contain a list > > > of filters used by consumers subscribed to that queue. However, for some > > > reason this list is updated in a very strange way. For one or two > > > consumers there are no CONSUMER_CREATED and CONSUMER_CLOSED core > > > notifications which PostOfficeImpl would receive and update the list > > > accordingly (also managing the list from outside of QueueInfo is quite > > > weird). > > > From the 3rd consumer the management messages start flowing, and here > > > comes the catch: The CONSUMER_CREATED message contains _AMQ_FilterString > > > = "" whereas the CONSUMER_CLOSED message contains AMQ_FilterString = > > > null. So the filterStrings List keeps filling up by empty strings because > > > these don’t get removed based on a null value. > > > > > > Jan > > > > > > From: Jan Šmucr<mailto:jan.sm...@aimtecglobal.com> > > > Sent: pátek 11. srpna 2023 9:43 > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > > > Subject: RE: Hunting memory leaks > > > > > > Hello all. > > > > > > I know it’s not ideal but the broker is doing just fine (except for the > > > leak issue of course). > > > > > > I’ve tried upgrading to 2.30.0 and the broker still ends up on its knees > > > given enough load and only a little heap. In my testing case I’ve limited > > > the heap size to 64 MiB so that I wouldn't have to wait for days for > > > things to happen, and also the consumer creation/disposal rate is > > > different to the production state. Here’s a very simple code which > > > manages to take down the 64 MiB broker in about 10 to 15 minutes on Java > > > 11 and recent Windows 10: > > > > > > final String queueName = "clouedi-kestra"; > > > final String filter = ""; > > > final Thread[] threads = new Thread[16]; > > > for (int i = 0; i < threads.length; i++) { > > > threads[i] = new Thread(() -> { > > > try ( > > > ServerLocator locator = > > > ActiveMQClient.createServerLocator("tcp://localhost:61616"); > > > ClientSessionFactory sf = locator.createSessionFactory(); > > > ClientSession session = sf.createSession(false, true); > > > ) { > > > while (!session.isClosed()) { > > > try (ClientConsumer consumer = > > > session.createConsumer(SimpleString.toSimpleString(queueName), > > > SimpleString.toSimpleString(filter), 0, 0, false)) { > > > consumer.receive(1); > > > } > > > } > > > } catch (Exception e) { > > > throw new RuntimeException(e); > > > } > > > }); > > > threads[i].start(); > > > } > > > for (Thread thread : threads) { > > > thread.join(); > > > } > > > > > > Big thanks for your help! > > > Jan > > > > > > From: Arthur Naseef<mailto:a...@amlinv.com> > > > Sent: čtvrtek 10. srpna 2023 21:11 > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > > > Subject: Re: Hunting memory leaks > > > > > > Creating a consumer only to consume 1 message is not ideal - there's a lot > > > of overhead and work on the broker side when consumers are created. > > > > > > With that said, since the consumer should be getting closed properly, that > > > should not cause a leak. > > > > > > So first, I would prioritize the version update. Second, I would consider > > > changing the use of consumers so they are longer-lived - preferrably only > > > being removed once the application needs to stop consuming. > > > > > > If there is a need to throttle and/or control threading and parallel > > > processing of messages, perhaps Camel would be a good fit. > > > > > > Hope this helps. > > > > > > Art > > > > > > > > > On Wed, Aug 9, 2023 at 10:44 PM Jan Šmucr <jan.sm...@aimtecglobal.com> > > > wrote: > > > > > > > Hello all. Thank you for your insights. > > > > > > > > > > > > * I’m using the core Java library. > > > > > > > > > > > > * Consumers are being created once per poll but reused if there are > > > > multiple inbound files to deal with. I create consumers like > > > > > > > > try (final consumer = createConsumer(session, params)) { > > > > > > > > // ... > > > > } > > > > > > > > so I expect them to be closed automatically. > > > > > > > > > > > > * I don’t use JMS, but the core sessions are used one per thread. > > > > The > > > > number of sessions opened and reported by Artemis doesn’t change over > > > > time. > > > > > > > > > > > > * I cannot reproduce the issue yet. It’s a production cluster, so > > > > today I’m going to set up my own playground. > > > > > > > > > > > > Jan > > > > > > > > From: Justin Bertram<mailto:jbert...@apache.org> > > > > Sent: středa 9. srpna 2023 17:41 > > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > > > > Subject: Re: Hunting memory leaks > > > > > > > > I echo Tim's recommendation to use the latest release, but I don't mean > > > > to > > > > say that will certainly resolve the problem. > > > > > > > > I can't say if you're doing anything wrong without more information. Can > > > > you answer the following questions? > > > > > > > > - What client library are you using? > > > > - How often are consumers being created? > > > > - Are consumers being closed properly once they are no longer needed? > > > > - Are JMS sessions being used concurrently from multiple threads? > > > > - Do you have a way to reproduce this that you can provide to me? A > > > > reproducer would make diagnosing this issue much simpler. > > > > > > > > Entries to the list of filter strings are added when a consumer is > > > > created > > > > and removed when a consumer is closed so at first glance it appears > > > > you're > > > > leaking consumers. > > > > > > > > > > > > Justin > > > > > > > > On Wed, Aug 9, 2023 at 7:07 AM Jan Šmucr <jan.sm...@aimtecglobal.com> > > > > wrote: > > > > > > > > > Hello. > > > > > I’m using a simple master-slave Artemis 2.26.0 cluster, and I’m > > > > > noticing > > > > > heap usage growing more and more each day no matter the throughput. > > > > There’s > > > > > about 670 sessions at the same time opened for producers and > > > > > consumers. > > > > > Consumers are polling queues on regular basis, some once a second > > > > (meaning > > > > > 1s timeout), some less often. This is by design and cannot be altered. > > > > All > > > > > client resources are being reused as much as possible. Usually > > > > > there’s a > > > > > thread pool and the threads have a session opened, and wait for tasks > > > > > to > > > > be > > > > > available to them. > > > > > It appears to me that the more consumers there is the faster the > > > > > server > > > > > heap depletes. > > > > > Now, I’m not very familiar with leak hunting apps, so all I have are > > > > > tiny > > > > > hints that it may have something to do with filter strings not being > > > > reused > > > > > and/or thrown away when not needed any more. I don’t know if I can > > > > > post a > > > > > screenshot here so I uploaded it here: https://snipboard.io/LHifUK.jpg > > > > > This is from a heap dump opened in JMC JOverflow plugin. > > > > > Is there something obvious that I’m doing wrong? Do you have any > > > > > clues on > > > > > what is going on here? > > > > > Thank you. > > > > > Jan. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Clebert Suconic > > > > -- > Clebert Suconic > -- Clebert Suconic
