If you can use Java 6, try the following setting: -XX:+HeapDumpOnOutOfMemoryError
(and -XX:HeapDumpPath if you want the dump to go somewhere specific) This will give you a heap dump which should help a _lot_ to diagnose the problem. I'd recommend checking out the Memory Analyzer plugin for Eclipse, or failing that, you can use "jhat" which comes with Java 6. cheers, j. On Fri, May 2, 2008 at 9:52 AM, Michele Mazzucco <[EMAIL PROTECTED]> wrote: > > On 2 May 2008, at 00:20, Filip Hanik - Dev Lists wrote: > > > > Michele Mazzucco wrote: > > > > > Filip, > > > > > > if you mean on the client side, I have a thread pool configured to > > > create max 100 threads. > > > > > > > > there could still be a thread leak in your application somewhere > > > > No, there's no leak on my client app. It's a very simple benchmark for my > server. BTW, how would you explain the error occurring on AMQ? > > > > > > > > > about -Xmx, that's the maximum amount of memory. Why are you telling > > > me that by decreasing that value there would be more space for > > > threads? > > > > > > > > cause, threads use memory, and that memory is not part of the Java Heap, > and the JVM reserves memory for your java heap based on your -Xmx setting. > > > > Oh, you mean the thread stack, don't you? I'll try to decrease the stack > size to see what happens, even if I don't think that's the root. > Instead, the problem might occur because every time the run() method is > invoked a new session is created (and destroyed). How does AMQ deal with > this?, I think a thread is associated to each session, isn't it ? > I'll try to store the session into a thread local. > > > > Michele > > > > > > > > take a look at the "Inside the Java Virtual Machine" webinar at > > http://www.covalent.net/services/training/webinars.html > > > > it's recorded, and you'll learn how java memory management works > > > > Filip > > > > > Michele > > > > > > > > > On Thu, May 1, 2008 at 7:50 PM, Filip Hanik - Dev Lists > > > <[EMAIL PROTECTED]> wrote: > > > > > > > > > > when it happens, do a thread dump to make sure you don't have a thread > leak > > > > (too many threads) > > > > > > > > then lower your -Xmx setting, this will make more room in the JVM for > > > > threads, > > > > > > > > Filip > > > > > > > > > > > > > > > > Michele Mazzucco wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > I'm facing the following error while using AMQ 5.0 > > > > > > > > > > ERROR efaultMessageListenerContainer - Setup of JMS message listener > > > > > invoker failed - trying to recover > > > > > java.lang.OutOfMemoryError: unable to create new native thread > > > > > at java.lang.Thread.start0(Native Method) > > > > > at java.lang.Thread.start(Thread.java:574) > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.thread.DedicatedTaskRunner.<init>(DedicatedTaskRunner.java:42) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:74) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQSessionExecutor.wakeup(ActiveMQSessionExecutor.java:92) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQMessageConsumer.start(ActiveMQMessageConsumer.java:993) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:222) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1004) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:948) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:861) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.createConsumer(AbstractPollingMessageListenerContainer.java:437) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.createListenerConsumer(AbstractPollingMessageListenerContainer.java:216) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:297) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:254) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:871) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:818) > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:595) > > > > > Exception in thread "DefaultMessageListenerContainer-29" > > > > > java.lang.OutOfMemoryError: unable to create new native thread > > > > > at java.lang.Thread.start0(Native Method) > > > > > at java.lang.Thread.start(Thread.java:574) > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportThreadSupport.doStart(TransportThreadSupport.java:43)ERROR > > > > > > > > > > > > > Service - Async error occurred: > > > > > java.lang.OutOfMemoryError: unable to create new native thread > > > > > java.lang.OutOfMemoryError: unable to create new native thread > > > > > at java.lang.Thread.start0(Native Method) > > > > > at java.lang.Thread.start(Thread.java:574) > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.broker.TransportConnection.processShutdown(TransportConnection.java:326) > > > > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.activemq.command.ShutdownInfo.visit(ShutdownInfo.java:35) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172) > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:595) > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:368) > > > > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:48) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:56) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:56) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:63) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:56) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:56) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:256) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:220) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:168) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:188) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.AbstractJmsListeningContainer.createSharedConnection(AbstractJmsListeningContainer.java:209) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:197) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:701) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:683) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:835) > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:595) > > > > > Exception in thread "ActiveMQ Transport Server: > tcp://localhost:61616" > > > > > java.lang.OutOfMemoryError: unable to create new native thread > > > > > at java.lang.Thread.start0(Native Method) > > > > > at java.lang.Thread.start(Thread.java:574) > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:160) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:213) > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:595) > > > > > ERROR Service - Async error occurred: > > > > > java.lang.OutOfMemoryError: unable to create new native thread > > > > > java.lang.OutOfMemoryError: unable to create new native thread > > > > > at java.lang.Thread.start0(Native Method) > > > > > at java.lang.Thread.start(Thread.java:574) > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.broker.TransportConnection.processShutdown(TransportConnection.java:326) > > > > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.activemq.command.ShutdownInfo.visit(ShutdownInfo.java:35) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172) > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:595) > > > > > INFO BrokerService - ActiveMQ Message Broker > > > > > (localhost, ID:giga06.ncl.ac.uk-41771-1209660890412-0:0) is shutting > > > > > down > > > > > INFO NetworkConnector - Network Connector > > > > > > > > > > > > > > > > > > > > [EMAIL PROTECTED] > > > > > > > > > > > > > Stopped > > > > > INFO TransportConnector - Connector openwire Stopped > > > > > INFO TransportConnector - Connector vm://localhost > Stopped > > > > > WARN ActiveMQConnection - Async exception with no > > > > > exception listener: > > > > > org.apache.activemq.transport.TransportDisposedIOException: Peer > > > > > (vm://localhost#1) disposed. > > > > > org.apache.activemq.transport.TransportDisposedIOException: Peer > > > > > (vm://localhost#1) disposed. > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:200) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) > > > > > > > > > > > > > INFO BrokerService - ActiveMQ JMS Message Broker > > > > > (localhost, ID:giga06.ncl.ac.uk-41771-1209660890412-0:0) stopped > > > > > INFO AdvisoryConsumer - Failed to send remove > command: > > > > > javax.jms.JMSException: Peer (vm://localhost#1) disposed. > > > > > javax.jms.JMSException: Peer (vm://localhost#1) disposed. > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1154) > > > > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.activemq.AdvisoryConsumer.dispose(AdvisoryConsumer.java:56) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnection.cleanup(ActiveMQConnection.java:1326) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnection.transportFailed(ActiveMQConnection.java:2027) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnection$4.run(ActiveMQConnection.java:1663) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:595) > > > > > Caused by: > org.apache.activemq.transport.TransportDisposedIOException: > > > > > Peer (vm://localhost#1) disposed. > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:87) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:59) > > > > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1152) > > > > > > > > > > > > > ... 7 more > > > > > > > > > > > > > > > My broker is running on a linux box with kernel 2.6 (I've read about > > > > > some similar issues when running on a kernel 2.4). > > > > > I'm using the following options when starting AMQ: > > > > > -Xms512m -Xmx1024m -Xmn500m > > > > > > > > > > I'm using the default configuration (no persistence). > > > > > I don't know whether it can help to diagnose the problem, my clients > > > > > are creating a lot of temporary queues (everything is cleaned up > after > > > > > use), but their lifetime is very short (a few seconds, they are just > > > > > used to receive a response message). > > > > > A similar problem occurs on my test client too (maybe it's related > to > > > > > the error on the broker), but not on the server. > > > > > > > > > > > > > > > Any help would be appreciated, > > > > > Michele > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
