Are you really limiting ram to 300 mb? Why so low if you are on a 8G box? Thanks, Tom
> On Jul 19, 2018, at 7:13 AM, art.licis <arturs.li...@gmail.com> wrote: > > Dear community, > > We've been evaluation AMQ which is based on version 5.11.0. Several tests in > staging environments which would have a big number of producers and > consumers ended up with OutOfMemory: GC Overhead Limit exceeded exception. > However, it looks like it's just one slow consumer that is causing this > memory problem (details below). > > Initially, we tried running AMQ with 4Gb of max heap, later with 8Gb but it > had the same outcome. > > Configuration: we have intentionally producer flow control turned off, and > expecting temp store to fill up before the producer is blocked > (non-persistent topic messages). I can simulate this with a simple and > controlled tests involving very fast producer, and several fast and some > slow consumers. With manual testing we achieve the desired result: when temp > store reaches the limit, the producer is blocked. > > We analyzed GC outputs and heap dump which we got upon the crash. Time spent > in GC keeps significantly increasing before the crash, and heap dump shows > the majority of memory occupied by PageFile object tree (lots of > PageFile$PageWrite objects, which indicate that too much data remains in > memory uncommitted to temp store; also related DataFileAppender$WriteBatch). > PageFile is the correct one, the one used in PListStoreImpl (i.e. temp store > for non-persistent messages). > > During my analysis I also discovered some bug that was fixed recently - > *AMQ-6815*. It looks like it affects the exact class family that could have > caused the memory leak. > > I'd be grateful for any help or suggestions. I'm currently preparing the > bigger test environment with many producers/consumers (and I'm also gonna > run Activemq in docker to limit I/O speed while I'm sure it wasn't the > issue), but the fact is that out of potentially thousands of > producers/consumers there was only one slow consumer. Eventually, we will > try running whole system against latest ActiveMQ, however, it will take lots > of arrangements and I'd like to see if I can make reproducable and > controlled test environment, and if there're some pitfalls in current setup. > > activemq.xml is below (the slow consumer was not the one listening to > virtual topic). And I will post more info if I'm able to solve the issue or > at least get more hints. > > Thank you. > -Art > > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd";> > > > <bean > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="properties"> > <bean class="io.fabric8.mq.fabric.ConfigurationProperties"/> > </property> > </bean> > > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="${broker-name}" dataDirectory="${data}" start="false" > restartAllowed="false"> > > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry topic=">" producerFlowControl="false"> > </policyEntry> > <policyEntry queue=">" producerFlowControl="false" > optimizedDispatch="true" > > </policyEntry> > </policyEntries> > </policyMap> > </destinationPolicy> > > <managementContext> > <managementContext createConnector="true" connectorPort="9001"/> > </managementContext> > > <ioExceptionHandler> > <defaultIOExceptionHandler ignoreNoSpaceErrors="false"/> > </ioExceptionHandler> > > <persistenceAdapter> > <kahaDB directory="${data}/kahadb"/> > </persistenceAdapter> > > <plugins> > </plugins> > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage limit="300 mb"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="100 gb"/> > </storeUsage> > <tempUsage> > <tempUsage limit="10 gb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > <transportConnectors> > <transportConnector name="openwire" > uri="tcp://0.0.0.0:0?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/> > </transportConnectors> > > <destinationInterceptors> > <virtualDestinationInterceptor> > <virtualDestinations> > <compositeQueue name="rpc.request.>" > forwardOnly="false"> > <forwardTo> > <topic > physicalName="rpc.request.virtual.notifications" /> > </forwardTo> > </compositeQueue> > <compositeQueue name="rpc.response.>" > forwardOnly="false"> > <forwardTo> > <topic > physicalName="rpc.response.virtual.notifications" /> > </forwardTo> > </compositeQueue> > </virtualDestinations> > </virtualDestinationInterceptor> > </destinationInterceptors> > > </broker> > > </beans> > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
