Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Hi Everyone, Can anyone please help me to give some solution on below error? I am getting this error after every 2-3 Hours. My activeMQ configuration is as follow. ActiveMQ configuration is as follow: ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS_MEMORY -Xms1g -Xmx1g" and activemq.xml configuration as follow: Tomcat Server Configration JVM Arguments: -XX:+UseParallelOldGC -Xms3g -Xmx3g -XX:NewRatio=3 -XX:PermSize=256m -XX:MaxPermSize=1024m Exception in thread "ajp-nio-8009-ClientPoller-1" Exception in thread "http-nio-8080-Acceptor-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "logback-1" java.lang.OutOfMemoryError: GC overhead limit exceeded 05-Jun-2018 07:45:09.213 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread [ContainerBackgroundProcessor[StandardEngine[Catalina]]] java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:3236) at java.util.zip.ZipCoder.getBytes(ZipCoder.java:89) at java.util.zip.ZipFile.getEntry(ZipFile.java:316) at java.util.jar.JarFile.getEntry(JarFile.java:240) at java.util.jar.JarFile.getJarEntry(JarFile.java:223) at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:98) at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256) at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281) at org.apache.catalina.webresources.Cache.getResource(Cache.java:62) at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:216) at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225) at org.apache.catalina.loader.WebappClassLoaderBase.modified(WebappClassLoaderBase.java:691) at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:343) at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:287) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5447) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1396) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1368) at java.lang.Thread.run(Thread.java:748) Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:3236) at java.util.zip.ZipCoder.getBytes(ZipCoder.java:89) at java.util.zip.ZipFile.getEntry(ZipFile.java:316) at java.util.jar.JarFile.getEntry(JarFile.java:240) at java.util.jar.JarFile.getJarEntry(JarFile.java:223) at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:98) at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256) at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281) at org.apache.catalina.webresources.Cache.getResource(Cache.java:62) at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:216) at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225) at org.apache.catalina.loader.WebappClassLoaderBase.modified(WebappClassLoaderBase.java:691) at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:343) at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:287) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5447) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1396) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1368) at java.lang.Thread.run(Thread.java:748) java.lang.OutOfMemoryError: GC overhead limit exceeded
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Hi @Tim Bain Can you please tell me how to enable the PFC in ActiveMQ? Can we pass some VM arguments to enable it? If Yes then which file should have those VM arguments.? Thank You -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
On Thu, Mar 15, 2018 at 1:45 AM, sathish.cre...@gmail.com < sathish.cre...@gmail.com> wrote: > Thanks a lot for your reply Tim, > > Below is the metrics details i grabbed from the server. > ActiveMQ: > 5.15.0 > non-persistent > Running consumer -> singleton -> onMessage() -> submit callables -> > callables send updated message to destination > messages -> 600/hour > messages size -> 8 byte > messages consume interval -> listener pools continuously as soon > as the > message arrives > JVM: > -Xms1g > -Xmx3g > Application: > -Xms4g > -Xmx4g > GCTYPE: > ParallelGC > > Its only consumer gives issues in Broker service side. > > 1. Yes. i do have to see how to enable PFC. But just a thought since its > only a consumer application do we need to to enable PFC? if so how do i > enable for consumer client side using programming? > Your original message in this thread said " I am having java.lang.OutOfMemoryError: GC overhead limit exceeded Error after a while from starting the ActiveMQ." Are you telling me that that sentence referred to a client rather than the broker? > 2. Also seeing below warning and does that makes sense even if am using non > persistence messages? > WARN BrokerService:2139 - Temporary Store limit is 51200 mb (current store > usage is 0 mb). The data directory: /temp only has 2000 mb of usable space. > - resetting to maximum available disk space: 2000 mb > Temp store is used to overflow to disk when the memory store gets full, so yes, it's relevant when using non-persistent messages. In fact, it's *only* relevant when using non-persistent messages. So that error message means that you can only have 2 GB more non-persistent messages than whatever fits in the memory store. > 3. The consumer design is like below, am i doing something wrong in the > design? > Consumer Main -> inject Consumer (init->activeMQ producer:consumer, > onMessage->call workerThread(consumer) ) -> workerThread (send message to > consumer, producer) > I have no idea what you're trying to describe here. Maybe take another shot at it, using sentences with enough words to convey unambiguous meaning, instead of arrow diagrams... > Expecting your valuable comments on this. > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Thanks a lot for your reply Tim, Below is the metrics details i grabbed from the server. ActiveMQ: 5.15.0 non-persistent Running consumer -> singleton -> onMessage() -> submit callables -> callables send updated message to destination messages -> 600/hour messages size -> 8 byte messages consume interval -> listener pools continuously as soon as the message arrives JVM: -Xms1g -Xmx3g Application: -Xms4g -Xmx4g GCTYPE: ParallelGC Its only consumer gives issues in Broker service side. 1. Yes. i do have to see how to enable PFC. But just a thought since its only a consumer application do we need to to enable PFC? if so how do i enable for consumer client side using programming? 2. Also seeing below warning and does that makes sense even if am using non persistence messages? WARN BrokerService:2139 - Temporary Store limit is 51200 mb (current store usage is 0 mb). The data directory: /temp only has 2000 mb of usable space. - resetting to maximum available disk space: 2000 mb 3. The consumer design is like below, am i doing something wrong in the design? Consumer Main -> inject Consumer (init->activeMQ producer:consumer, onMessage->call workerThread(consumer) ) -> workerThread (send message to consumer, producer) Expecting your valuable comments on this. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
1. http://activemq.apache.org/producer-flow-control.html, and yes, you should enable it. You should protect the broker (by preventing messages from over-filling its heap) first and foremost, and PFC does that. 2. When you're out of memory, it tends to manifest itself all over the place. There's no direct causal relationship between the specific calls that fail and the running out of memory; all those calls tell you is that they're using memory, not that they're using an unreasonable amount of it or failing to free it. Things that are relevant in any OOM situation: what version of ActiveMQ, what max heap size are you using, what GC strategy are you using, and what's the usage pattern for your broker (how many messages/second, how many queues/topics, persistent or non-persistent messages, how big are the messages, how long do the messages sit on the broker before being consumed, etc.)? I'd start with enabling PFC, and allocating more heap to the broker process if that's an option, and then I'd spend time characterizing what the expected worst-case memory usage is based on how many messages you expect to have in broker memory (non-persistent messages, plus persistent messages paged in for dispatch to consumers, see http://activemq.apache.org/message-cursors.html for an explanation) in the worst-case scenario you need to support. It's possible that you simply don't have the heap sized large enough to support the use-case you're trying to perform. Tim On Wed, Mar 7, 2018 at 2:06 AM, sathish.cre...@gmail.com < sathish.cre...@gmail.com> wrote: > Hi artnaseef, > > i am facing same issue in my startup log after 5 days, similar to this, > > Exception in thread "ActiveMQ InactivityMonitor Worker" Exception in thread > "pool-3-thread-121233" Exception in thread "pool-1-thread-1" Exception in > thread "pool-3-thread-432434" java.lang.OutOfMemoryError: GC overhead limit > exceeded > java.lang.OutOfMemoryError: GC overhead limit exceeded > > Let me know if below options will help, > 1. should i enable producer flow control? if so how do i enable it? > 2. getting this exception also along with this , > > Exception in thread "ActiveMQ Transport Server: > ssl://ip:port?socket.needClientAuth=true" java.lang.OutOfMemoryError: GC > overhead limit exceeded > at sun.security.ssl.InputRecord.(InputRecord.java:93) > at sun.security.ssl.AppInputStream.(AppInputStream.java:50) > at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:640) > at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:524) > at > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:343) > at > org.apache.activemq.transport.tcp.TcpTransportServer. > doRunWithServerSocket(TcpTransportServer.java:403) > at > org.apache.activemq.transport.tcp.TcpTransportServer.run( > TcpTransportServer.java:325) > at java.lang.Thread.run(Thread.java:748) > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Hi artnaseef, i am facing same issue in my startup log after 5 days, similar to this, Exception in thread "ActiveMQ InactivityMonitor Worker" Exception in thread "pool-3-thread-121233" Exception in thread "pool-1-thread-1" Exception in thread "pool-3-thread-432434" java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded Let me know if below options will help, 1. should i enable producer flow control? if so how do i enable it? 2. getting this exception also along with this , Exception in thread "ActiveMQ Transport Server: ssl://ip:port?socket.needClientAuth=true" java.lang.OutOfMemoryError: GC overhead limit exceeded at sun.security.ssl.InputRecord.(InputRecord.java:93) at sun.security.ssl.AppInputStream.(AppInputStream.java:50) at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:640) at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:524) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:343) at org.apache.activemq.transport.tcp.TcpTransportServer.doRunWithServerSocket(TcpTransportServer.java:403) at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:325) at java.lang.Thread.run(Thread.java:748) -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Check first for slow consumption. Is producer-flow-control enabled on the broker? And, if it is, what is the per-queue memory limit and the overall system limit. The most common cause of Out Of Memory on ActiveMQ is a broker without producer-flow-control and slow consumption, which mimics a memory leak. -- View this message in context: http://activemq.2283324.n4.nabble.com/java-lang-OutOfMemoryError-GC-overhead-limit-exceeded-tp4707387p4707909.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
i am using ParallelGC (the default garbage collector) here is some information : XX:InitialHeapSize=262927424 -XX:MaxHeapSize=4206838784 -XX:+PrintCommandLineFlags -XX:+UseCompressedOops -XX:+UseParallelGC -- View this message in context: http://activemq.2283324.n4.nabble.com/java-lang-OutOfMemoryError-GC-overhead-limit-exceeded-tp4707387p4707519.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
First, for a question about GC problems, you should probably tell us what GC strategy you're using, and why you chose it. Second, you made the claim that "to increase the memory to 2Gb ... only delaying the issue from happening," which implies that you believe you're running out of memory because of a memory leak rather than because you sized your JVM too small. Do you have any evidence of such a leak or that you've properly sized your JVM and run out of memory, or are you just making an assumption? I've spent a couple months performance testing the 5.8.0 broker's memory usage with non-persistent messages that are consumed immediately, and I can tell you for certain that there's no memory leak in the code paths executed by that specific scenario (in that version). It's always possible that there's a leak in some other code that my test wasn't executing, but even that is somewhat unlikely given that no one but you has complained about it. So although it's possible that there's a leak, I think it's more likely that either you have more live objects than will fit into the amount of heap you've given to your JVM (in which case increasing the heap is the solution, and a real solution), or you're using the CMS GC strategy and you're hitting the Old Gen fragmentation problem. Tim On Feb 10, 2016 10:52 AM, "ealiass" wrote: > Hi All, > > I am having java.lang.OutOfMemoryError: GC overhead limit exceeded Error > after a while from starting the ActiveMQ on my system (2-3 weeks) start > seeing this issue on production. > i had to restart the my ActiveMQ every while on production so i might lose > some messages coming in the time while the queue is restarting. > > we are using the version:5.11.1 > we have 1Gb of memory for the queue on production > there was similar issue on the forum and the answer was to increase the > memory to 2Gb, but this is not a real solution since we only delaying the > issue from happening > > find below the logs i got > > > org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire,connectionViewType=remoteAddress,connectionName=tcp_//172.16.19.0_40176\n","stream":"stdout","time":"2016-01-18T10:40:55.455379802Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52528 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:40:58.87142185Z"} > {"log":"ERROR | Could not accept connection : java.lang.Exception: > java.lang.OutOfMemoryError: GC overhead limit > exceeded\n","stream":"stdout","time":"2016-01-18T10:41:00.229514014Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40222 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.994358557Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40195 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.99828402Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40193 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:06.424638043Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40374 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:09.107570289Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34082 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:11.89354178Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34072 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:13.548763654Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52575 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.788697766Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52628 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.795035469Z"} > {"log":"ERROR | Could not accept connection : java.lang.Exception: > java.lang.OutOfMemoryError: GC overhead limit > exceeded\n","stream":"stdout","time":"2016-01-18T10:41:17.821256534Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40341 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:17.824982456Z"} > {"log":"Exception in thread \"ActiveMQ InactivityMonitor Worker\" > java.lang.OutOfMemoryError: GC overhead limit > exceeded\n","stream":"stderr","time":"2016-01-18T10:41:22.453110848Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34040 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:22.459674904Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40300 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:26.945482864Z"} > {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52777 failed: > > java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:30.77306023Z"} > {"log":"Exception in thread \"ActiveMQ BrokerService[localhost] > Task-72110\" > java.lang.OutOfMemoryError