I mean PR. It would punch me in the face sooner or later anyway. I do use my consumers. It's just that I am sending and receiving large files between various systems, many of which run in Bash and the scripts must not fail if Artemis happens to be offline, so there's quite a lot of things to optimize. There are multiple jobs on multiple servers, each polling for different things in one or more queues. Having M jobs, one thread per N cores, and the need for receiving O files in parallel from one or more sources made me think twice about what to discard and what to reuse. Even now it's fairly complicated. Still, one broker was handling it pretty well with a nice uptime. But then I incorporated an empty filter set of consumers, and my pipes started leaking. 🙂
Jan Zasláno z Outlooku pro Android<https://aka.ms/AAb9ysg> ________________________________ From: Clebert Suconic <clebert.suco...@gmail.com> Sent: Saturday, August 12, 2023 5:35:59 PM To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: Hunting memory leaks Also. You should be using your consumers. On Sat, Aug 12, 2023 at 11:35 AM Clebert Suconic <clebert.suco...@gmail.com> wrote: > You mean you are sending a PR or you will use null instead of “”? > > On Sat, Aug 12, 2023 at 11:11 AM Jan Šmucr <jan.sm...@aimtecglobal.com> > wrote: > >> Ok, I'll fix it then. My Jira at work will be happy for another Done >> task. 😁 >> >> Jan >> >> Zasláno z Outlooku pro Android<https://aka.ms/AAb9ysg> >> ________________________________ >> From: Clebert Suconic <clebert.suco...@gmail.com> >> Sent: Saturday, August 12, 2023 5:05:49 PM >> To: users@activemq.apache.org <users@activemq.apache.org> >> Subject: Re: Hunting memory leaks >> >> Just for a future reference, this is where the JMS layer protects this >> from happening: >> >> >> https://github.com/apache/activemq-artemis/blob/064018a3e9a1ba39ddbee0bbddfed3e7fccab89c/artemis-jms-client/src/main/java/org/apache/activemq/artemis/jms/client/ActiveMQSession.java#L592-L594 >> >> On Sat, Aug 12, 2023 at 11:03 AM Clebert Suconic >> <clebert.suco...@gmail.com> wrote: >> > >> > This only happened because you used the Core API directly, and passed >> > a "" as the filter String. >> > >> > >> > In the JMS layer, there's a check to replace "" by null and that would >> > leave that out. >> > >> > there's a check to add and remove existing filters from the >> > selectors.. but in your case this is creating a leak of "" strings. >> > >> > >> > >> > You should instead pass in null instead of "" for now. >> > >> > >> > We could add a layer of protection on the client and on the server >> > replacing "" by null both client and server. >> > >> > >> > As for a test... We could add a test under ./tests/leak-tests... >> > counting the number of SimpleStrings with CheckLeak (look at other >> > tests) and then making sure it won't go beyond a certain limit. >> > >> > >> > (Do you want to send the PR yourself?) >> > >> > >> > >> > I'm not going to rush for a fix now as you can just use null instead >> > of "" on your filter string and this leak won't happen (I already >> > tested it). >> > >> > >> > let me know if you want to send the PR.. as I don't want to duplicate >> > your efforts. If you're not fixing it just let me know and I will do >> > it on monday. >> > >> > >> > Thanks for this ! >> > >> > On Sat, Aug 12, 2023 at 10:53 AM Jan Šmucr <jan.sm...@aimtecglobal.com> >> wrote: >> > > >> > > I'll leave it up to you. If you're busy, I'll have created a PR by >> Monday too. 🙂 And as a bonus, I'll get better acquaintanted with the >> Artemis code. >> > > >> > > Jan >> > > >> > > ________________________________ >> > > Od: Clebert Suconic <clebert.suco...@gmail.com> >> > > Odesláno: sobota, srpna 12, 2023 4:48:08 odp. >> > > Komu: users@activemq.apache.org <users@activemq.apache.org> >> > > Předmět: Re: Hunting memory leaks >> > > >> > > The leak is because you are creating a consumer within the same >> > > session over and over: >> > > >> > > >> https://github.com/apache/activemq-artemis/blob/064018a3e9a1ba39ddbee0bbddfed3e7fccab89c/artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/PostOfficeImpl.java#L410-L420 >> > > >> > > >> > > Do you want to raise the JIRA for this? I should have a fix by monday. >> > > >> > > >> > > If you keep your consumer open instead of open / close it all the >> > > time this won't happen. But I should have a fix by monday. >> > > >> > > On Fri, Aug 11, 2023 at 12:24 PM Clebert Suconic >> > > <clebert.suco...@gmail.com> wrote: >> > > > >> > > > I highly recommend you using check-leak.. you would have found >> what's >> > > > leaking already. >> > > > >> > > > https://github.com/check-leak/check-leak >> > > > >> > > > java -jar check-leak-0.10.jar remote --pid <PID> --report >> > > > <reportoutput> --sleep 5000 >> > > > >> > > > ( I suggest using 5 seconds for your test) >> > > > >> > > > >> > > > >> > > > I would even write a unit-test for memory-leaks. >> > > > >> > > > On Fri, Aug 11, 2023 at 10:06 AM Jan Šmucr < >> jan.sm...@aimtecglobal.com> wrote: >> > > > > >> > > > > So I’m getting a bit closer. The leak is in PostOfficeImpl and >> QueueInfo. QueueInfo contains the filterStrings List which appears to >> contain a list of filters used by consumers subscribed to that queue. >> However, for some reason this list is updated in a very strange way. For >> one or two consumers there are no CONSUMER_CREATED and CONSUMER_CLOSED core >> notifications which PostOfficeImpl would receive and update the list >> accordingly (also managing the list from outside of QueueInfo is quite >> weird). >> > > > > From the 3rd consumer the management messages start flowing, and >> here comes the catch: The CONSUMER_CREATED message contains >> _AMQ_FilterString = "" whereas the CONSUMER_CLOSED message contains >> AMQ_FilterString = null. So the filterStrings List keeps filling up by >> empty strings because these don’t get removed based on a null value. >> > > > > >> > > > > Jan >> > > > > >> > > > > From: Jan Šmucr<mailto:jan.sm...@aimtecglobal.com> >> > > > > Sent: pátek 11. srpna 2023 9:43 >> > > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> >> > > > > Subject: RE: Hunting memory leaks >> > > > > >> > > > > Hello all. >> > > > > >> > > > > I know it’s not ideal but the broker is doing just fine (except >> for the leak issue of course). >> > > > > >> > > > > I’ve tried upgrading to 2.30.0 and the broker still ends up on >> its knees given enough load and only a little heap. In my testing case I’ve >> limited the heap size to 64 MiB so that I wouldn't have to wait for days >> for things to happen, and also the consumer creation/disposal rate is >> different to the production state. Here’s a very simple code which manages >> to take down the 64 MiB broker in about 10 to 15 minutes on Java 11 and >> recent Windows 10: >> > > > > >> > > > > final String queueName = "clouedi-kestra"; >> > > > > final String filter = ""; >> > > > > final Thread[] threads = new Thread[16]; >> > > > > for (int i = 0; i < threads.length; i++) { >> > > > > threads[i] = new Thread(() -> { >> > > > > try ( >> > > > > ServerLocator locator = >> ActiveMQClient.createServerLocator("tcp://localhost:61616"); >> > > > > ClientSessionFactory sf = >> locator.createSessionFactory(); >> > > > > ClientSession session = sf.createSession(false, >> true); >> > > > > ) { >> > > > > while (!session.isClosed()) { >> > > > > try (ClientConsumer consumer = >> session.createConsumer(SimpleString.toSimpleString(queueName), >> SimpleString.toSimpleString(filter), 0, 0, false)) { >> > > > > consumer.receive(1); >> > > > > } >> > > > > } >> > > > > } catch (Exception e) { >> > > > > throw new RuntimeException(e); >> > > > > } >> > > > > }); >> > > > > threads[i].start(); >> > > > > } >> > > > > for (Thread thread : threads) { >> > > > > thread.join(); >> > > > > } >> > > > > >> > > > > Big thanks for your help! >> > > > > Jan >> > > > > >> > > > > From: Arthur Naseef<mailto:a...@amlinv.com> >> > > > > Sent: čtvrtek 10. srpna 2023 21:11 >> > > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> >> > > > > Subject: Re: Hunting memory leaks >> > > > > >> > > > > Creating a consumer only to consume 1 message is not ideal - >> there's a lot >> > > > > of overhead and work on the broker side when consumers are >> created. >> > > > > >> > > > > With that said, since the consumer should be getting closed >> properly, that >> > > > > should not cause a leak. >> > > > > >> > > > > So first, I would prioritize the version update. Second, I would >> consider >> > > > > changing the use of consumers so they are longer-lived - >> preferrably only >> > > > > being removed once the application needs to stop consuming. >> > > > > >> > > > > If there is a need to throttle and/or control threading and >> parallel >> > > > > processing of messages, perhaps Camel would be a good fit. >> > > > > >> > > > > Hope this helps. >> > > > > >> > > > > Art >> > > > > >> > > > > >> > > > > On Wed, Aug 9, 2023 at 10:44 PM Jan Šmucr < >> jan.sm...@aimtecglobal.com> >> > > > > wrote: >> > > > > >> > > > > > Hello all. Thank you for your insights. >> > > > > > >> > > > > > >> > > > > > * I’m using the core Java library. >> > > > > > >> > > > > > >> > > > > > * Consumers are being created once per poll but reused if >> there are >> > > > > > multiple inbound files to deal with. I create consumers like >> > > > > > >> > > > > > try (final consumer = createConsumer(session, params)) { >> > > > > > >> > > > > > // ... >> > > > > > } >> > > > > > >> > > > > > so I expect them to be closed automatically. >> > > > > > >> > > > > > >> > > > > > * I don’t use JMS, but the core sessions are used one per >> thread. The >> > > > > > number of sessions opened and reported by Artemis doesn’t >> change over time. >> > > > > > >> > > > > > >> > > > > > * I cannot reproduce the issue yet. It’s a production >> cluster, so >> > > > > > today I’m going to set up my own playground. >> > > > > > >> > > > > > >> > > > > > Jan >> > > > > > >> > > > > > From: Justin Bertram<mailto:jbert...@apache.org> >> > > > > > Sent: středa 9. srpna 2023 17:41 >> > > > > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> >> > > > > > Subject: Re: Hunting memory leaks >> > > > > > >> > > > > > I echo Tim's recommendation to use the latest release, but I >> don't mean to >> > > > > > say that will certainly resolve the problem. >> > > > > > >> > > > > > I can't say if you're doing anything wrong without more >> information. Can >> > > > > > you answer the following questions? >> > > > > > >> > > > > > - What client library are you using? >> > > > > > - How often are consumers being created? >> > > > > > - Are consumers being closed properly once they are no longer >> needed? >> > > > > > - Are JMS sessions being used concurrently from multiple >> threads? >> > > > > > - Do you have a way to reproduce this that you can provide to >> me? A >> > > > > > reproducer would make diagnosing this issue much simpler. >> > > > > > >> > > > > > Entries to the list of filter strings are added when a consumer >> is created >> > > > > > and removed when a consumer is closed so at first glance it >> appears you're >> > > > > > leaking consumers. >> > > > > > >> > > > > > >> > > > > > Justin >> > > > > > >> > > > > > On Wed, Aug 9, 2023 at 7:07 AM Jan Šmucr < >> jan.sm...@aimtecglobal.com> >> > > > > > wrote: >> > > > > > >> > > > > > > Hello. >> > > > > > > I’m using a simple master-slave Artemis 2.26.0 cluster, and >> I’m noticing >> > > > > > > heap usage growing more and more each day no matter the >> throughput. >> > > > > > There’s >> > > > > > > about 670 sessions at the same time opened for producers and >> consumers. >> > > > > > > Consumers are polling queues on regular basis, some once a >> second >> > > > > > (meaning >> > > > > > > 1s timeout), some less often. This is by design and cannot be >> altered. >> > > > > > All >> > > > > > > client resources are being reused as much as possible. >> Usually there’s a >> > > > > > > thread pool and the threads have a session opened, and wait >> for tasks to >> > > > > > be >> > > > > > > available to them. >> > > > > > > It appears to me that the more consumers there is the faster >> the server >> > > > > > > heap depletes. >> > > > > > > Now, I’m not very familiar with leak hunting apps, so all I >> have are tiny >> > > > > > > hints that it may have something to do with filter strings >> not being >> > > > > > reused >> > > > > > > and/or thrown away when not needed any more. I don’t know if >> I can post a >> > > > > > > screenshot here so I uploaded it here: >> https://snipboard.io/LHifUK.jpg >> > > > > > > This is from a heap dump opened in JMC JOverflow plugin. >> > > > > > > Is there something obvious that I’m doing wrong? Do you have >> any clues on >> > > > > > > what is going on here? >> > > > > > > Thank you. >> > > > > > > Jan. >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > -- >> > > > Clebert Suconic >> > > >> > > >> > > >> > > -- >> > > Clebert Suconic >> > > >> > >> > >> > -- >> > Clebert Suconic >> >> >> >> -- >> Clebert Suconic >> > -- > Clebert Suconic > -- Clebert Suconic
