Gary,
In my config file I have multicast in both these entries. Which one you
think possibly causing this memory leak?. We plan to have failover broker so
we may need multicast discovery?.
<networkConnectors>
<networkConnector name="default" uri="multicast://default"/>
</networkConnectors>
<transportConnectors>
<transportConnector name="default" uri="tcp://localhost:61616"
discoveryUri="multicast://default"/>
</transportConnectors>
Gary Tully wrote:
>
> the config looks ok, just one thought, do you need multicast discovery on
> your transport connector? I wonder if it is possible you are seeing
> https://issues.apache.org/activemq/browse/AMQ-2283 which resulted in an
> leak
> of discovery threads? Removing the discoveryUri will work around it.
>
> 2009/10/13 tejeswara <tejesw...@gmail.com>
>
>>
>> Here is our activemq.xml. As you can see it is pretty much same as the
>> template version we get from activemq package:
>> <beans
>> xmlns="http://www.springframework.org/schema/beans";
>> xmlns:amq="http://activemq.apache.org/schema/core";
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> http://activemq.apache.org/schema/core
>> http://activemq.apache.org/schema/core/activemq-core.xsd
>> http://activemq.apache.org/camel/schema/spring
>> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";>
>> <bean
>>
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> <property name="locations">
>>
>> <value>file:///${activemq.base}/conf/credentials.properties</value>
>> </property>
>> </bean>
>> <broker xmlns="http://activemq.apache.org/schema/core";
>> brokerName="localhost" dataDirectory="${activemq.base}/data">
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry queue=">" memoryLimit="50mb"/>
>> <policyEntry topic=">" memoryLimit="5mb">
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>> <managementContext>
>> <managementContext createConnector="true"
>> connectorPort="11099"/>
>> </managementContext>
>> <networkConnectors>
>> <networkConnector name="default-nc"
>> uri="multicast://default"/>
>> </networkConnectors>
>> <persistenceAdapter>
>> <amqPersistenceAdapter syncOnWrite="false"
>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>> </persistenceAdapter>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="200 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="1 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>> <transportConnectors>
>> <transportConnector name="openwire"
>> uri="tcp://localhost:61616"
>> discoveryUri="multicast://default"/>
>> </transportConnectors>
>> </broker>
>> <camelContext id="camel"
>> xmlns="http://activemq.apache.org/camel/schema/spring";>
>> <package>org.foo.bar</package>
>> <route>
>> <from uri="activemq:example.A"/>
>> <to uri="activemq:example.B"/>
>> </route>
>> </camelContext>
>> <bean id="activemq"
>> class="org.apache.activemq.camel.component.ActiveMQComponent" >
>> <property name="connectionFactory">
>> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
>> <property name="brokerURL"
>> value="vm://localhost?create=false&waitForStart=10000" />
>> <property name="userName" value="${activemq.username}"/>
>> <property name="password" value="${activemq.password}"/>
>> </bean>
>> </property>
>> </bean>
>> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0";>
>> <connectors>
>> <nioConnector port="11098"/>
>> </connectors>
>> <handlers>
>> <webAppContext contextPath="/admin"
>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>> <webAppContext contextPath="/demo"
>> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
>> <webAppContext contextPath="/fileserver"
>> resourceBase="${activemq.base}/webapps/fileserver" logUrlOnStart="true"/>
>> </handlers>
>> </jetty>
>> </beans>
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > Can you post your activemq.xml? It looks like you need to restrict the
>> > memory usage of the broker and have it spool messages to disk when the
>> > memory limits are reached. See some additional detail at
>> >
>> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
>> > *
>> >
>> > *
>> > 2009/10/12 tejeswara <tejesw...@gmail.com>
>> >
>> >>
>> >> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I see
>> >> below
>> >> error in logs.
>> >> Note that, below values are set in bin/activemq in out test lab:
>> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> >> ACTIVEMQ_OPTS="-Xmx512M
>> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
>> >> fi
>> >>
>> >> I tried to stress test broker by creating 200 queues and put 5000
>> >> messages
>> >> (each about 1500 bytes size)
>> >> in each of these queues. However I could not see any error. Below
>> error
>> >> occured when broker is idle.
>> >> From logs, only activity during this idle period I see is 'DEBUG
>> >> InactivityMonitor - 30177 ms elapsed since last read
>> check'
>> >>
>> >> Can someone help in pointing out what is going wrong?
>> >>
>> >> Note that, OutOfMemoryError is seen in our production even when heap
>> size
>> >> is
>> >> bumped up to 1024 and
>> >> dedicated task runner is set to false. This error usvally occurs after
>> 2
>> >> days broker running without restart.
>> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
>> >> ACTIVEMQ_OPTS="-Xmx1024M
>> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
>> >> fi
>> >>
>> >>
>> >> Stack Trace:->
>> >>
>> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
>> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
>> >> JVMDUMP010I Heap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
>> >> JVMDUMP010I Java Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
>> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError".
>> >> Exception in thread "MulticastDiscovery: null"
>> >> java.lang.OutOfMemoryError:
>> >> heap allocation failed
>> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> >> at
>> >>
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
>> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
>> >> JVMDUMP016E Aborting: Cannot create file
>> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
>> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
>> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> >> JVMDUMP010I Java Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
>> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError".
>> >> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
>> >> java.lang.OutOfMemoryError: heap allocation failed
>> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
>> >> at
>> >>
>> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
>> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> JVMDUMP006I Processing Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError" - Please Wait.
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
>> >> JVMDUMP007I JVM Requesting Heap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
>> >> JVMDUMP016E Aborting: Cannot create file
>> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
>> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
>> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
>> >> JVMDUMP010I Java Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
>> >> JVMDUMP013I Processed Dump Event "uncaught", detail
>> >> "java/lang/OutOfMemoryError".
>> >> Exception in thread "InactivityMonitor WriteCheck"
>> >> java.lang.OutOfMemoryError
>> >> at java.security.AccessController.getProtectionDomains(Native
>> >> Method)
>> >> at
>> >> java.security.AccessController.getContext(AccessController.java:132)
>> >> at java.lang.Thread.initialize(Thread.java:347)
>> >> at java.lang.Thread.<init>(Thread.java:278)
>> >> at java.lang.Thread.<init>(Thread.java:170)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
>> >> at
>> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
>> >> at
>> >>
>> >>
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> >> at java.util.TimerThread.mainLoop(Timer.java:527)
>> >> at java.util.TimerThread.run(Timer.java:477)
>> >> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
>> >> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
>> >> at
>> >>
>> >>
>> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
>> >> at
>> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
>> >> at sun.rmi.transport.Transport$1.run(Transport.java:168)
>> >> at
>> >> java.security.AccessController.doPrivileged(AccessController.java:275)
>> >> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
>> >> at
>> >>
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
>> >> at
>> >>
>> >>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
>> >> at
>> >>
>> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
>> >> at
>> >>
>> >>
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
>> >> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
>> >> at
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
>> >> at
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
>> >> at
>> >>
>> >>
>> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> Exception in thread "RMI LeaseChecker" java.lang.NoClassDefFoundError:
>> >> java.rmi.server.Unreferenced
>> >> at sun.rmi.transport.Target.refSetRemove(Target.java:320)
>> >> at sun.rmi.transport.Target.vmidDead(Target.java:444)
>> >> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
>> >> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
>> >> at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
>> >> at java.lang.Thread.run(Thread.java:803)
>> >> Unhandled exception
>> >> Type=Segmentation error vmState=0x00053fff
>> >> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
>> >> Error_Value=00000000
>> >> Signal_Code=00000001
>> >> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
>> >> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
>> >> ECX=08B7DF40 EDX=00000000
>> >> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
>> >> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
>> >> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
>> >> Module_base_address=00F09000
>> >>
>> >>
>> >>
>> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
>> >> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
>> >> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
>> >> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
>> >> JVMDUMP007I JVM Requesting System Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
>> >> JVMDUMP012E Error in System Dump: cannot find core file, check "ulimit
>> >> -c"
>> >> is set high enough
>> >> JVMDUMP007I JVM Requesting Snap Dump using
>> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
>> >> JVMDUMP010I Snap Dump written to
>> >> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
>> >> JVMDUMP007I JVM Requesting Java Dump using
>> >> '/tmp/javacore.20091011.220932.16581.txt'
>> >> NULL
>> >>
>> ------------------------------------------------------------------------
>> >> 0SECTION TITLE subcomponent dump routine
>> >> NULL ===============================
>> >> 1TISIGINFO Dump Event "gpf" (00002000) received
>> >> 1TIDATETIME Date: 2009/10/11 at 22:10:17
>> >> 1TIFILENAME Javacore filename:
>> >> /tmp/javacore.20091011.220932.16581.txt
>> >> NULL
>> >>
>> ------------------------------------------------------------------------
>> >> 0SECTION GPINFO subcomponent dump routine
>> >> NULL ================================
>> >> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp
>> >> 2XHCPUS Processors -
>> >> 3XHCPUARCH Architecture : x86
>> >> 3XHNUMCPUS How Many : 8
>> >> NULL
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context:
http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25935327.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
