Hey @Justin Bertram <jbert...@apache.org> Undestood. Can you please explain to me why I am getting the following suspected memory leaks in the Memory Analyzer tool (for a taken memory heap)? Is this considered normal?
Problem Suspect 1 1,006 instances of *org.apache.activemq.artemis.core.server.impl.QueueImpl*, loaded by *java.net.URLClassLoader @ 0x82721c20* occupy *37,566,648 (33.76%)* bytes. Most of these instances are referenced from one instance of *java.util.concurrent.ConcurrentHashMap$Node[]*, loaded by *<system class loader>*, which occupies *40,496 (0.04%)* bytes. The instance is referenced by *org.apache.activemq.artemis.utils.collections.ConcurrentHashSet @ 0x81cf40b0*, loaded by *java.net.URLClassLoader @ 0x82721c20*. Thread *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread @ 0x81fed740 Apache ActiveMQ Artemis libaio poller* has a local variable or reference to *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory @ 0x81cf68a8* which is on the shortest path to *java.util.concurrent.ConcurrentHashMap$Node[2048] @ 0x81cf4100*. The thread *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread @ 0x81fed740 Apache ActiveMQ Artemis libaio poller* keeps local variables with total size *200 (0.00%)* bytes. The stacktrace of this Thread is available. See stacktrace. See stacktrace with involved local variables. *Keywords* - org.apache.activemq.artemis.core.server.impl.QueueImpl - java.net.URLClassLoader - java.util.concurrent.ConcurrentHashMap$Node[] - org.apache.activemq.artemis.utils.collections.ConcurrentHashSet Leak Suspects 2,000 instances of *org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl*, loaded by *java.net.URLClassLoader @ 0x82721c20* occupy *16,823,208 (15.12%)* bytes. Most of these instances are referenced from one instance of *java.util.HashMap$Node[]*, loaded by *<system class loader>*, which occupies *40,208 (0.04%)* bytes. The instance is referenced by *org.apache.activemq.artemis.core.server.cluster.ClusterManager @ 0x81ff0148*, loaded by *java.net.URLClassLoader @ 0x82721c20*. Thread *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread @ 0x81fee598 activemq-failure-check-thread* has a local variable or reference to *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl @ 0x82757548* which is on the shortest path to *java.util.HashMap$Node[2048] @ 0x8206b9f8*. The thread *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread @ 0x81fee598 activemq-failure-check-thread* keeps local variables with total size *448 (0.00%)* bytes. The stacktrace of this Thread is available. See stacktrace. See stacktrace with involved local variables. *Keywords* - org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl - java.net.URLClassLoader - java.util.HashMap$Node[] - org.apache.activemq.artemis.core.server.cluster.ClusterManager On Fri, Dec 6, 2024 at 12:01 AM Clebert Suconic <clebert.suco...@gmail.com> wrote: > I hoped you could give me some piece of code isolating the issue, in a > way I could execute a script and see exactly what you're seeing. > > On Wed, Dec 4, 2024 at 11:00 AM Dragan J <draga...@flipside.team> wrote: > > > > I have already sent the output of the memory suspect analysis before. > > 1,254 instances of > > > “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > > loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 > > (73.68%) bytes. > > > > I created the new 2 test servers with 1,000 CoreBridges between them, > > killed the Java process for the Artemis server, and monitored the memory > > heap on the client over time (through JConsole). A memory heap increase > was > > noticed again. > > The archived servers (both the client and server) along with a screenshot > > of the memory heap (G1 Old Gen) can be found at: > > > https://drive.google.com/file/d/1TBMUV31JbTvZwDF0P2teX0PSCIA_C1kn/view?usp=drive_link > > > https://drive.google.com/file/d/1iJ1O9PLlL_8XnIbMvE7mTBrz26mhce0I/view?usp=sharing > > > https://drive.google.com/file/d/1-eItVKsubvQXnbyNQI2Jj88hh1kJK0lM/view?usp=sharing > > > > Please note that the installation of Artemis 2.38 should be in the /opt > > folder. > > > > Thanks, > > Dragan > > > > > > On Wed, Dec 4, 2024 at 7:02 AM Clebert Suconic < > clebert.suco...@gmail.com> > > wrote: > > > > > Do you have a test showing the issue ? > > > > > > > > > check Leak has an interface to count instances. If you provide a way > to > > > reproduce the issue (a test with embedded server would be best) I will > help > > > out. > > > > > > Clebert Suconic > > > > > > > > > On Tue, Dec 3, 2024 at 9:03 AM Dragan J <draga...@flipside.team> > wrote: > > > > > > > Hi guys, > > > > > > > > Do you have any updates on this? > > > > > > > > Thanks, > > > > Dragan > > > > > > > > On Tue, Nov 26, 2024 at 4:27 PM Dragan J <draga...@flipside.team> > wrote: > > > > > > > > > @justin, please try rebooting the server with an Artemis broker > (the > > > > > source) that connects to another broker (the destination), which > must > > > run > > > > > on a different server. > > > > > The brokers exchange messages between them, but the volume is very > low > > > > > since this is a development environment. > > > > > A memory leak is observed a few hours later on the destination > server > > > > > (running Artemis 2.38). > > > > > > > > > > @Clebert The provided check-leak code is not straightforward to > > > > > implement. Could you please refer me to some instructions on how > to do > > > > it? > > > > > > > > > > > > > > > > > > > > On Fri, Nov 22, 2024 at 3:00 PM Dragan J <draga...@flipside.team> > > > wrote: > > > > > > > > > >> @Clebert Unfortunately, the business requirement is to use > > > CoreBridges. > > > > >> @Justin We tested with 1000 CoreBridges, and a memory leak > remains. > > > > >> Here is "the problem suspect" from the Memory Analyzer (a heap > dump > > > > >> analyzed): > > > > >> > > > > >> > > > > >> 1,254 instances of > > > > >> > > > > > > > > “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > > > > >> loaded by “java.net.URLClassLoader @ 0x400000000” occupy > 527,112,720 > > > > >> (73.68%) bytes. > > > > >> > > > > >> Biggest instances: > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40d81e128 - 17,669,864 (2.47%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409926c00 - 17,643,464 (2.47%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x411b0b320 - 17,624,664 (2.46%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x401fdb948 - 17,614,232 (2.46%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x4099afd28 - 17,606,640 (2.46%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409895e10 - 17,599,608 (2.46%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40d9e35e0 - 17,597,288 (2.46%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409b1a978 - 17,581,512 (2.46%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409969ad8 - 17,579,656 (2.46%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40d9e5088 - 17,559,432 (2.45%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x410fa8d80 - 17,552,744 (2.45%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x404db4d38 - 17,547,832 (2.45%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x4197b64f0 - 17,529,272 (2.45%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409a03058 - 17,521,432 (2.45%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x411b55168 - 17,495,208 (2.45%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40b0b3c30 - 17,481,480 (2.44%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x4084fe4b0 - 17,478,696 (2.44%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x4195a5dc8 - 17,476,840 (2.44%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x419725708 - 17,472,008 (2.44%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409a441c8 - 17,453,176 (2.44%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x404d89060 - 17,434,424 (2.44%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40f875d80 - 17,425,144 (2.44%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x4102071f8 - 17,408,104 (2.43%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40d9e34f8 - 17,383,576 (2.43%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409926318 - 17,374,568 (2.43%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x409a7cde8 - 17,318,888 (2.42%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40aadd150 - 17,285,208 (2.42%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40d81ba08 - 17,273,880 (2.41%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x40070d068 - 17,223,032 (2.41%) bytes. > > > > >> > > > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> @ 0x41bea4538 - 17,146,936 (2.40%) bytes. > > > > >> > > > > >> Most of these instances are referenced from one instance of > > > > >> > > > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl”, > > > > >> loaded by “java.net.URLClassLoader @ 0x400000000”, which occupies > > > 7,008 > > > > >> (0.00%) bytes. > > > > >> > > > > >> Thread > > > > >> > > > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > > > > >> @ 0x405ffd540 activemq-failure-check-thread” has a local variable > or > > > > >> reference to > > > > >> > > > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > > > >> @ 0x401a42658” which is on the shortest path to > > > > >> > > > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > > > >> @ 0x401a42658”. The thread > > > > >> > > > > > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > > > > >> @ 0x405ffd540 activemq-failure-check-thread keeps local variables > with > > > > >> total size 328 (0.00%) bytes. > > > > >> > > > > >> The stacktrace of this Thread is available. See stacktrace. See > > > > >> stacktrace with involved local variables. > > > > >> > > > > >> Keywords > > > > >> > > > > > > > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > > >> java.net.URLClassLoader > > > > >> > > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > > > >> > > > > >> Thanks, > > > > >> Dragan > > > > >> > > > > >> > > > > >> On Tue, Nov 19, 2024 at 12:13 PM Dragan J <draga...@flipside.team > > > > > > wrote: > > > > >> > > > > >>> Hey Justin, > > > > >>> > > > > >>> Here are the answers: > > > > >>> - Are all the bridges configured the same way? How exactly are > they > > > > >>> configured? > > > > >>> Yes, they are configured the same way. > > > > >>> The CoreBridge configuration is attached to > > > > >>> > > > > > > > > https://drive.google.com/file/d/1VEwj1MoB2tbcCp8R4XeQTzi80Q0dQ606/view?usp=sharing > > > > >>> > > > > >>> - Is the bridge target available when you see this increase > happen? > > > > >>> The bridges are connected from the client side to the server > (with > > > the > > > > >>> 2.38 Artemis version) > > > > >>> - Are the bridges actually moving messages when you see the > increase > > > or > > > > >>> are they idle? > > > > >>> Most of the time, messages are idle. > > > > >>> - Are 3,500 bridges actually required to trigger the increase? Do > > > you, > > > > >>> for > > > > >>> example, not see the increase if you use 3,000 bridges? What > about > > > > 1,000 > > > > >>> or > > > > >>> 100? > > > > >>> We tested with 3500 CoreBridges in the development environment, > and I > > > > >>> should ask for permission to close some connections. > > > > >>> If you require it, I would ask for that kind of permission. > > > > >>> > > > > >>> Thanks, > > > > >>> Dragan > > > > >>> > > > > >>> On Mon, Nov 18, 2024 at 2:43 PM Dragan J <draga...@flipside.team > > > > > > wrote: > > > > >>> > > > > >>>> Can you access the screenshots? > > > > >>>> > > > > >>>> On Thu, Nov 14, 2024 at 10:18 AM Dragan J > <draga...@flipside.team> > > > > >>>> wrote: > > > > >>>> > > > > >>>>> Sorry, a typo. It's 3,500 CoreBridges > > > > >>>>> > > > > >>>>> On Thu, Nov 14, 2024 at 10:14 AM Dragan J > <draga...@flipside.team> > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>>> @Justin The broker (version 2.38) communicates with other > brokers > > > > >>>>>> (version 2.37) through 35,000 CoreBridges. There was minimal > > > message > > > > >>>>>> exchange between them. > > > > >>>>>> Please find the two graphs (from earlier) attached on my > Google > > > > >>>>>> Drive. Please let me know if you have an alternative way to > share > > > > files. > > > > >>>>>> > > > > >>>>>> > > > > > > > > https://drive.google.com/file/d/1Gy8XQi153Fi5Zm9gUdsP_co8fSuh1HjL/view?usp=sharing > > > > >>>>>> > > > > >>>>>> > > > > > > > > https://drive.google.com/file/d/1_Fbg8OmWhlsOKfalfIEwKxGIl97a1kQ5/view?usp=sharing > > > > >>>>>> > > > > >>>>>> @Arthur When I tried to perform a GC from JConsole I got the > > > > >>>>>> following: > > > > >>>>>> > > > > >>>>>> Exception in thread "MemoryPanel.gc" > java.lang.SecurityException: > > > > >>>>>> User not authorized to access operation: gc > > > > >>>>>> > > > > >>>>> > > > > > > > > > > > -- > Clebert Suconic > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@activemq.apache.org > For additional commands, e-mail: dev-h...@activemq.apache.org > For further information, visit: https://activemq.apache.org/contact > > >
