Hi Bruce,
Thanks for the response. Looking at JVM options is definitely a way to
go but I have a suspicion that that would be treating the symptoms. We
really don't have many destinations. The setup should _theoretically_
keep the number of connections low (pooledconnection factory for
producers, CACHE_CONSUMER in the consumers) but it looks like
something is generating a _lot_ connections regardless. The
FailoverTransport.doReconnect messages are suspicious but I don't know
for sure that they're bad...
regards,
Jamie.
On Thu, Oct 1, 2009 at 2:48 PM, Bruce Snyder <bruce.sny...@gmail.com> wrote:
> On Thu, Oct 1, 2009 at 5:13 AM, Jamie McCrindle
> <jamiemccrin...@gmail.com> wrote:
>> Hi All,
>>
>> I've just got hit by an OutOfMemoryError: unable to create new native
>> thread issue. I've switched UseDedicatedTaskRunner to false which
>> appears to have resolved the issue for now but I have a feeling I'm
>> just masking a deeper problem.
>>
>> I have the following set up:
>>
>> 2 ActiveMQ brokers configured as a network of brokers
>> Several applications that use
>> - a Spring JmsTemplate with a PooledConnectionFactory
>> - a Spring DefaultMessageListenerContainer that uses a regular
>> ConnectionFactory. The DefaultMessageListenerContainers are set to
>> CACHE_CONSUMER
>> - a static failover broker URL for both the JmsTemplate and listener
>> container to the two brokers
>>
>> In the applications themselves, I'm getting this:
>>
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:574)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:455)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:863)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.wakeup(PooledTaskRunner.java:76)
>> at
>> org.apache.activemq.transport.failover.FailoverTransport.reconnect(FailoverTransport.java:536)
>> at
>> org.apache.activemq.transport.failover.FailoverTransport.start(FailoverTransport.java:239)
>> at
>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>> at
>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>> at
>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:260)
>> at
>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:224)
>> at
>> org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:172)
>> at
>> org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:125)
>> at
>> org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:110)
>> at
>> org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:81)
>> at
>> org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
>> at
>> org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:461)
>> at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:539)
>>
>> In the brokers I'm getting this:
>>
>> INFO | jvm 1 | 2009/09/30 12:28:41 | Exception in thread
>> "ActiveMQ Transport Server Thread Hand
>> ler: tcp://localhost:41414" java.lang.OutOfMemoryError: unable to
>> create new native thread
>> INFO | jvm 1 | 2009/09/30 12:28:41 | at
>> java.lang.Thread.start0(Native Method)
>> INFO | jvm 1 | 2009/09/30 12:28:41 | at
>> java.lang.Thread.start(Thread.java:574)
>> INFO | jvm 1 | 2009/09/30 12:28:41 | at
>> org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:221)
>> INFO | jvm 1 | 2009/09/30 12:28:41 | at
>> org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:409)
>> INFO | jvm 1 | 2009/09/30 12:28:41 | at
>> org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:341)
>> INFO | jvm 1 | 2009/09/30 12:28:41 | at
>> java.lang.Thread.run(Thread.java:595)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | Exception in thread
>> "DefaultMessageListenerContainer-80896" java.lang.OutOfMemoryError:
>> unable to create new native thread
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> java.lang.Thread.start0(Native Method)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> java.lang.Thread.start(Thread.java:574)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> org.springframework.core.task.SimpleAsyncTaskExecutor.doExecute(SimpleAsyncTaskExecutor.java:157)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:145)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:126)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> org.springframework.jms.listener.DefaultMessageListenerContainer.doRescheduleTask(DefaultMessageListenerContainer.java:598)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> org.springframework.jms.listener.AbstractJmsListeningContainer.rescheduleTaskIfNecessary(AbstractJmsListeningContainer.java:509)
>> INFO | jvm 1 | 2009/09/30 12:59:27 | at
>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:918)
>>
>> I'm also getting a lot of these in the logs:
>>
>> FailoverTransport.doReconnect | Successfully connected to tcp://xxxxxxxx:xxxx
>>
>> Other information:
>>
>> We've got a similar set up in our load testing environment and our
>> other production environment and neither of them appear to have the
>> same problem (yet?).
>> I've read the JmsTemplate gotchas and outofmemory pages on the
>> ActiveMQ site. I think I'm within the guidelines...
>> We're using Spring 2.5.6
>> We aren't using a transaction manager
>>
>> any ideas?
>
> Not using the DedicatedTaskRunner means that less threads are created
> which will indeed help to alleviate the problem. But you should look
> at the JVM settings being used, especially the ThreadStackSize for the
> error you were receiving.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)u8...@4vyy9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>
