If you are using large messages please use the latest version anyway. As I had done some fixes with large messages in Some edge cases with redeliveries.
We should have a 2.31 release soon. (As I’m looking forward to release the CLI improvements :) ) On Sat, Aug 12, 2023 at 1:38 PM Jan Šmucr <jan.sm...@aimtecglobal.com> wrote: > I mean PR. It would punch me in the face sooner or later anyway. > I do use my consumers. It's just that I am sending and receiving large > files between various systems, many of which run in Bash and the scripts > must not fail if Artemis happens to be offline, so there's quite a lot of > things to optimize. There are multiple jobs on multiple servers, each > polling for different things in one or more queues. Having M jobs, one > thread per N cores, and the need for receiving O files in parallel from one > or more sources made me think twice about what to discard and what to > reuse. Even now it's fairly complicated. Still, one broker was handling it > pretty well with a nice uptime. But then I incorporated an empty filter set > of consumers, and my pipes started leaking. 🙂 > > Jan > > Zasláno z Outlooku pro Android<https://aka.ms/AAb9ysg> > ________________________________ > From: Clebert Suconic <clebert.suco...@gmail.com> > Sent: Saturday, August 12, 2023 5:35:59 PM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: Hunting memory leaks > > Also. You should be using your consumers. > > > On Sat, Aug 12, 2023 at 11:35 AM Clebert Suconic < > clebert.suco...@gmail.com> > wrote: > > > You mean you are sending a PR or you will use null instead of “”? > > > > On Sat, Aug 12, 2023 at 11:11 AM Jan Šmucr <jan.sm...@aimtecglobal.com> > > wrote: > > > >> Ok, I'll fix it then. My Jira at work will be happy for another Done > >> task. 😁 > >> > >> Jan > >> > >> Zasláno z Outlooku pro Android<https://aka.ms/AAb9ysg> > >> ________________________________ > >> From: Clebert Suconic <clebert.suco...@gmail.com> > >> Sent: Saturday, August 12, 2023 5:05:49 PM > >> To: users@activemq.apache.org <users@activemq.apache.org> > >> Subject: Re: Hunting memory leaks > >> > >> Just for a future reference, this is where the JMS layer protects this > >> from happening: > >> > >> > >> > https://github.com/apache/activemq-artemis/blob/064018a3e9a1ba39ddbee0bbddfed3e7fccab89c/artemis-jms-client/src/main/java/org/apache/activemq/artemis/jms/client/ActiveMQSession.java#L592-L594 > >> > >> On Sat, Aug 12, 2023 at 11:03 AM Clebert Suconic > >> <clebert.suco...@gmail.com> wrote: > >> > > >> > This only happened because you used the Core API directly, and passed > >> > a "" as the filter String. > >> > > >> > > >> > In the JMS layer, there's a check to replace "" by null and that would > >> > leave that out. > >> > > >> > there's a check to add and remove existing filters from the > >> > selectors.. but in your case this is creating a leak of "" strings. > >> > > >> > > >> > > >> > You should instead pass in null instead of "" for now. > >> > > >> > > >> > We could add a layer of protection on the client and on the server > >> > replacing "" by null both client and server. > >> > > >> > > >> > As for a test... We could add a test under ./tests/leak-tests... > >> > counting the number of SimpleStrings with CheckLeak (look at other > >> > tests) and then making sure it won't go beyond a certain limit. > >> > > >> > > >> > (Do you want to send the PR yourself?) > >> > > >> > > >> > > >> > I'm not going to rush for a fix now as you can just use null instead > >> > of "" on your filter string and this leak won't happen (I already > >> > tested it). > >> > > >> > > >> > let me know if you want to send the PR.. as I don't want to duplicate > >> > your efforts. If you're not fixing it just let me know and I will do > >> > it on monday. > >> > > >> > > >> > Thanks for this ! > >> > > >> > On Sat, Aug 12, 2023 at 10:53 AM Jan Šmucr < > jan.sm...@aimtecglobal.com> > >> wrote: > >> > > > >> > > I'll leave it up to you. If you're busy, I'll have created a PR by > >> Monday too. 🙂 And as a bonus, I'll get better acquaintanted with the > >> Artemis code. > >> > > > >> > > Jan > >> > > > >> > > ________________________________ > >> > > Od: Clebert Suconic <clebert.suco...@gmail.com> > >> > > Odesláno: sobota, srpna 12, 2023 4:48:08 odp. > >> > > Komu: users@activemq.apache.org <users@activemq.apache.org> > >> > > Předmět: Re: Hunting memory leaks > >> > > > >> > > The leak is because you are creating a consumer within the same > >> > > session over and over: > >> > > > >> > > > >> > https://github.com/apache/activemq-artemis/blob/064018a3e9a1ba39ddbee0bbddfed3e7fccab89c/artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/PostOfficeImpl.java#L410-L420 > >> > > > >> > > > >> > > Do you want to raise the JIRA for this? I should have a fix by > monday. > >> > > > >> > > > >> > > If you keep your consumer open instead of open / close it all the > >> > > time this won't happen. But I should have a fix by monday. > >> > > > >> > > On Fri, Aug 11, 2023 at 12:24 PM Clebert Suconic > >> > > <clebert.suco...@gmail.com> wrote: > >> > > > > >> > > > I highly recommend you using check-leak.. you would have found > >> what's > >> > > > leaking already. > >> > > > > >> > > > https://github.com/check-leak/check-leak > >> > > > > >> > > > java -jar check-leak-0.10.jar remote --pid <PID> --report > >> > > > <reportoutput> --sleep 5000 > >> > > > > >> > > > ( I suggest using 5 seconds for your test) > >> > > > > >> > > > > >> > > > > >> > > > I would even write a unit-test for memory-leaks. > >> > > > > >> > > > On Fri, Aug 11, 2023 at 10:06 AM Jan Šmucr < > >> jan.sm...@aimtecglobal.com> wrote: > >> > > > > > >> > > > > So I’m getting a bit closer. The leak is in PostOfficeImpl and > >> QueueInfo. QueueInfo contains the filterStrings List which appears to > >> contain a list of filters used by consumers subscribed to that queue. > >> However, for some reason this list is updated in a very strange way. For > >> one or two consumers there are no CONSUMER_CREATED and CONSUMER_CLOSED > core > >> notifications which PostOfficeImpl would receive and update the list > >> accordingly (also managing the list from outside of QueueInfo is quite > >> weird). > >> > > > > From the 3rd consumer the management messages start flowing, and > >> here comes the catch: The CONSUMER_CREATED message contains > >> _AMQ_FilterString = "" whereas the CONSUMER_CLOSED message contains > >> AMQ_FilterString = null. So the filterStrings List keeps filling up by > >> empty strings because these don’t get removed based on a null value. > >> > > > > > >> > > > > Jan > >> > > > > > >> > > > > From: Jan Šmucr<mailto:jan.sm...@aimtecglobal.com> > >> > > > > Sent: pátek 11. srpna 2023 9:43 > >> > > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > >> > > > > Subject: RE: Hunting memory leaks > >> > > > > > >> > > > > Hello all. > >> > > > > > >> > > > > I know it’s not ideal but the broker is doing just fine (except > >> for the leak issue of course). > >> > > > > > >> > > > > I’ve tried upgrading to 2.30.0 and the broker still ends up on > >> its knees given enough load and only a little heap. In my testing case > I’ve > >> limited the heap size to 64 MiB so that I wouldn't have to wait for days > >> for things to happen, and also the consumer creation/disposal rate is > >> different to the production state. Here’s a very simple code which > manages > >> to take down the 64 MiB broker in about 10 to 15 minutes on Java 11 and > >> recent Windows 10: > >> > > > > > >> > > > > final String queueName = "clouedi-kestra"; > >> > > > > final String filter = ""; > >> > > > > final Thread[] threads = new Thread[16]; > >> > > > > for (int i = 0; i < threads.length; i++) { > >> > > > > threads[i] = new Thread(() -> { > >> > > > > try ( > >> > > > > ServerLocator locator = > >> ActiveMQClient.createServerLocator("tcp://localhost:61616"); > >> > > > > ClientSessionFactory sf = > >> locator.createSessionFactory(); > >> > > > > ClientSession session = sf.createSession(false, > >> true); > >> > > > > ) { > >> > > > > while (!session.isClosed()) { > >> > > > > try (ClientConsumer consumer = > >> session.createConsumer(SimpleString.toSimpleString(queueName), > >> SimpleString.toSimpleString(filter), 0, 0, false)) { > >> > > > > consumer.receive(1); > >> > > > > } > >> > > > > } > >> > > > > } catch (Exception e) { > >> > > > > throw new RuntimeException(e); > >> > > > > } > >> > > > > }); > >> > > > > threads[i].start(); > >> > > > > } > >> > > > > for (Thread thread : threads) { > >> > > > > thread.join(); > >> > > > > } > >> > > > > > >> > > > > Big thanks for your help! > >> > > > > Jan > >> > > > > > >> > > > > From: Arthur Naseef<mailto:a...@amlinv.com> > >> > > > > Sent: čtvrtek 10. srpna 2023 21:11 > >> > > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > >> > > > > Subject: Re: Hunting memory leaks > >> > > > > > >> > > > > Creating a consumer only to consume 1 message is not ideal - > >> there's a lot > >> > > > > of overhead and work on the broker side when consumers are > >> created. > >> > > > > > >> > > > > With that said, since the consumer should be getting closed > >> properly, that > >> > > > > should not cause a leak. > >> > > > > > >> > > > > So first, I would prioritize the version update. Second, I > would > >> consider > >> > > > > changing the use of consumers so they are longer-lived - > >> preferrably only > >> > > > > being removed once the application needs to stop consuming. > >> > > > > > >> > > > > If there is a need to throttle and/or control threading and > >> parallel > >> > > > > processing of messages, perhaps Camel would be a good fit. > >> > > > > > >> > > > > Hope this helps. > >> > > > > > >> > > > > Art > >> > > > > > >> > > > > > >> > > > > On Wed, Aug 9, 2023 at 10:44 PM Jan Šmucr < > >> jan.sm...@aimtecglobal.com> > >> > > > > wrote: > >> > > > > > >> > > > > > Hello all. Thank you for your insights. > >> > > > > > > >> > > > > > > >> > > > > > * I’m using the core Java library. > >> > > > > > > >> > > > > > > >> > > > > > * Consumers are being created once per poll but reused if > >> there are > >> > > > > > multiple inbound files to deal with. I create consumers like > >> > > > > > > >> > > > > > try (final consumer = createConsumer(session, params)) { > >> > > > > > > >> > > > > > // ... > >> > > > > > } > >> > > > > > > >> > > > > > so I expect them to be closed automatically. > >> > > > > > > >> > > > > > > >> > > > > > * I don’t use JMS, but the core sessions are used one per > >> thread. The > >> > > > > > number of sessions opened and reported by Artemis doesn’t > >> change over time. > >> > > > > > > >> > > > > > > >> > > > > > * I cannot reproduce the issue yet. It’s a production > >> cluster, so > >> > > > > > today I’m going to set up my own playground. > >> > > > > > > >> > > > > > > >> > > > > > Jan > >> > > > > > > >> > > > > > From: Justin Bertram<mailto:jbert...@apache.org> > >> > > > > > Sent: středa 9. srpna 2023 17:41 > >> > > > > > To: users@activemq.apache.org<mailto: > users@activemq.apache.org> > >> > > > > > Subject: Re: Hunting memory leaks > >> > > > > > > >> > > > > > I echo Tim's recommendation to use the latest release, but I > >> don't mean to > >> > > > > > say that will certainly resolve the problem. > >> > > > > > > >> > > > > > I can't say if you're doing anything wrong without more > >> information. Can > >> > > > > > you answer the following questions? > >> > > > > > > >> > > > > > - What client library are you using? > >> > > > > > - How often are consumers being created? > >> > > > > > - Are consumers being closed properly once they are no longer > >> needed? > >> > > > > > - Are JMS sessions being used concurrently from multiple > >> threads? > >> > > > > > - Do you have a way to reproduce this that you can provide to > >> me? A > >> > > > > > reproducer would make diagnosing this issue much simpler. > >> > > > > > > >> > > > > > Entries to the list of filter strings are added when a > consumer > >> is created > >> > > > > > and removed when a consumer is closed so at first glance it > >> appears you're > >> > > > > > leaking consumers. > >> > > > > > > >> > > > > > > >> > > > > > Justin > >> > > > > > > >> > > > > > On Wed, Aug 9, 2023 at 7:07 AM Jan Šmucr < > >> jan.sm...@aimtecglobal.com> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hello. > >> > > > > > > I’m using a simple master-slave Artemis 2.26.0 cluster, and > >> I’m noticing > >> > > > > > > heap usage growing more and more each day no matter the > >> throughput. > >> > > > > > There’s > >> > > > > > > about 670 sessions at the same time opened for producers and > >> consumers. > >> > > > > > > Consumers are polling queues on regular basis, some once a > >> second > >> > > > > > (meaning > >> > > > > > > 1s timeout), some less often. This is by design and cannot > be > >> altered. > >> > > > > > All > >> > > > > > > client resources are being reused as much as possible. > >> Usually there’s a > >> > > > > > > thread pool and the threads have a session opened, and wait > >> for tasks to > >> > > > > > be > >> > > > > > > available to them. > >> > > > > > > It appears to me that the more consumers there is the faster > >> the server > >> > > > > > > heap depletes. > >> > > > > > > Now, I’m not very familiar with leak hunting apps, so all I > >> have are tiny > >> > > > > > > hints that it may have something to do with filter strings > >> not being > >> > > > > > reused > >> > > > > > > and/or thrown away when not needed any more. I don’t know if > >> I can post a > >> > > > > > > screenshot here so I uploaded it here: > >> https://snipboard.io/LHifUK.jpg > >> > > > > > > This is from a heap dump opened in JMC JOverflow plugin. > >> > > > > > > Is there something obvious that I’m doing wrong? Do you have > >> any clues on > >> > > > > > > what is going on here? > >> > > > > > > Thank you. > >> > > > > > > Jan. > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Clebert Suconic > >> > > > >> > > > >> > > > >> > > -- > >> > > Clebert Suconic > >> > > > >> > > >> > > >> > -- > >> > Clebert Suconic > >> > >> > >> > >> -- > >> Clebert Suconic > >> > > -- > > Clebert Suconic > > > -- > Clebert Suconic > -- Clebert Suconic
