I forgot you mentioned to do that before I posted the sa project to the list. Here is the link to the Jira I created:
https://issues.apache.org/activemq/browse/SM-1519 It's fully replicable with the project I attached. On Sat, Aug 23, 2008 at 9:47 AM, Gert Vanthienen <[EMAIL PROTECTED]>wrote: > Ryan, > > Do you think you could build a simple SA to reproduce the issue (e.g. with > a quartz SU or a Camel time that sends a lot of exchanges to the topic)? If > so, could you raise a JIRA and attach the SA so we can look into this? > > > Regards, > > Gert > > Ryan Moquin wrote: > >> I changed my broker in my activemq.xml in servicemix like this: >> >> <!-- ActiveMQ JMS Broker configuration --> >> <amq:broker id="broker" depends-on="jmxServer" useShutdownHook="false" >> persistent="false"> >> >> and then started it up. It went to 1444 and then stopped processing >> anything. So my guess is that that doesn't help either. I think some of >> my >> other services are able to keep sender so it seems like the producer flow >> control is kicking in rather than my messages being expired like I want. >> >> On Fri, Aug 22, 2008 at 7:43 AM, Gert Vanthienen >> <[EMAIL PROTECTED]>wrote: >> >> >> >>> Ryan, >>> >>> Odd... Just to be sure we are not looking at something completely >>> different >>> here, could you try to disable persistence on the broker alltogether to >>> ensure that this fixes the problem? From the stacktraces, it looks like >>> it >>> is the SEDA flow itself that is hanging here (waiting for thread to >>> become >>> available after a sendSync to somewhere perhaps?) >>> >>> Regards, >>> >>> Gert >>> >>> >>> Ryan Moquin wrote: >>> >>> >>> >>>> Just a little bit more. So everything I find on this issue, seems to >>>> indicate I guess that since the JMS delivery mode is persistent by >>>> default, >>>> you should turn off persistence if you don't want a producer to block. >>>> So >>>> since I don't want my service to block, nor do I want the messages >>>> persistent, I configured my jms provider like this: >>>> >>>> <jms:provider service="not:notification-jms-service" >>>> endpoint="notification" >>>> pubSubDomain="true" >>>> destinationName="topic.notication" >>>> connectionFactory="#connectionFactory" >>>> deliveryMode="1" >>>> timeToLive="1000"/> >>>> >>>> As far as I can tell (since it's not documented what the values of >>>> deliveryMode are), this should keep my producer from blocking, but it >>>> still >>>> does after it hits a little over a 1000 messages. Is there a way to >>>> turn >>>> off persistence in my producers that work? I have not found a way to >>>> make >>>> this happen. >>>> >>>> On Tue, Aug 19, 2008 at 7:54 PM, Ryan Moquin <[EMAIL PROTECTED]> >>>> wrote: >>>> >>>> >>>> >>>> >>>> >>>>> Maybe this is the better way to ask this as I try to rationalize this >>>>> behavior in my head: >>>>> >>>>> If you have a service unit that sends messages to a topic created by a >>>>> servicemix-jms provider endpoint that has no consumers, shouldn't every >>>>> message sent to it, be discarded? I'm not really sure I actually know >>>>> what >>>>> the problem is, since all my configuration attempts to fix it have >>>>> failed. >>>>> All I want at this point is any way possible to prevent the system from >>>>> deadlocking. I'd be happy going back to exceptions being thrown all >>>>> the >>>>> time instead. At one point with Servicemix 3.2.2, we've processed lots >>>>> and >>>>> lots of messages (eventually Servicemix ran out of memory and crashed, >>>>> no >>>>> clue as to why that happened consistently every 2 hours, but that's >>>>> another >>>>> issue, I'm guessing because ActiveMQ never discards any messages). We >>>>> haven't changed any of the servicemix configurations since then, so I'm >>>>> not >>>>> sure exactly why we can only process a small amount before Servicemix >>>>> will >>>>> hang from ActiveMQ blocking. >>>>> >>>>> I'm very puzzled at this point. >>>>> >>>>> >>>>> On Tue, Aug 19, 2008 at 10:33 AM, Ryan Moquin <[EMAIL PROTECTED] >>>>> >>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>>> Hello, >>>>>> >>>>>> So, due to my service units hanging under certain conditions when I >>>>>> try >>>>>> to >>>>>> send a JMS message using an injected DeliveryChannel, I switched to >>>>>> using a >>>>>> ServicemixClient I look up out of the servicemix JNDI context. This >>>>>> is >>>>>> done >>>>>> by several of my deployed services that are doing various pieces of >>>>>> work. >>>>>> I'm now noticing that while switching to a separate ServicemixClient >>>>>> has >>>>>> fixed some of my issues, I'm now seeing certain services that send >>>>>> messages >>>>>> to a JMS service unit to send out of the syste, are also now >>>>>> deadlocking >>>>>> around the same point and therefore my service halts. I did a thread >>>>>> dump >>>>>> to verify this and I see A LOT of threads that are waiting on a >>>>>> specific >>>>>> lock with id: 0x1abeba80. You can see one of these below: >>>>>> >>>>>> "pool-flow.seda.servicemix-jms-thread-1157" prio=6 tid=0x3cee2400 >>>>>> nid=0x1a00 waiting on condition [0 >>>>>> x3869f000..0x3869fa98] >>>>>> java.lang.Thread.State: WAITING (parking) >>>>>> at sun.misc.Unsafe.park(Native Method) >>>>>> - parking to wait for <0x1abeba80> (a >>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer >>>>>> $ConditionObject) >>>>>> at >>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue >>>>>> dSynchronizer.java:1925) >>>>>> at >>>>>> >>>>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp >>>>>> l.java:663) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>>> at java.lang.Thread.run(Thread.java:619) >>>>>> >>>>>> My service itself is also locking on the same id, while trying to send >>>>>> a >>>>>> message to the JMS consumer (I think is the right term): >>>>>> >>>>>> "QuartzScheduler_Worker-7" prio=6 tid=0x3a2f1400 nid=0x1050 waiting on >>>>>> condition [0x37e2e000..0x37e2 >>>>>> fc18] >>>>>> java.lang.Thread.State: WAITING (parking) >>>>>> at sun.misc.Unsafe.park(Native Method) >>>>>> - parking to wait for <0x1abeba80> (a >>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer >>>>>> $ConditionObject) >>>>>> at >>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue >>>>>> dSynchronizer.java:1925) >>>>>> at >>>>>> >>>>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp >>>>>> l.java:663) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec >>>>>> utor.java:1737) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88 >>>>>> ) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3 >>>>>> 95) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431 >>>>>> ) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jms.multiplexing.MultiplexingProviderProcessor.process(Multiplexing >>>>>> ProviderProcessor.java:143) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:538) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:4 >>>>>> 90) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.common.BaseLifeCycle.onMessageExchange(BaseLifeCycle.java:46) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp >>>>>> l.java:610) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec >>>>>> utor.java:1737) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88 >>>>>> ) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3 >>>>>> 95) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431 >>>>>> ) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:14 >>>>>> 7) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:21 >>>>>> 1) >>>>>> >>>>>> So my question is, is activemq freezing up for some reason? All my >>>>>> problems I have, seem to somehow stem to activemq in various ways, >>>>>> usually >>>>>> all my freeze ups seem to be attributed to ActiveMQ freezing up for >>>>>> one >>>>>> reason or another. If the above case is that ActiveMQ is freezing up >>>>>> (it >>>>>> happens over a period of time, I thought I read that ActiveMQ will >>>>>> block >>>>>> when a topic or Queue fills up, which makes no sense to me in regards >>>>>> to >>>>>> a >>>>>> topic, since I thought that a topic doesn't hold messages unless they >>>>>> are >>>>>> marked durable), are there settings I can use to keep this from >>>>>> happening? >>>>>> There is no way I can ever trust this sort of system in production >>>>>> unless I >>>>>> can be sure that Servicemix/ActiveMQ isn't going to lock up on me >>>>>> after >>>>>> a >>>>>> couple hours. >>>>>> >>>>>> In case you want to see how I'm using the ServicemixClient, I'm not >>>>>> doing >>>>>> anything overly special, but here it is: >>>>>> >>>>>> if (client == null) { >>>>>> ClientFactory factory = (ClientFactory) new >>>>>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME); >>>>>> client = factory.createClient(); >>>>>> } >>>>>> >>>>>> if (destinationQname == null) { >>>>>> destinationQname = new QName(getDestinationNamespace(), >>>>>> getDestinationService()); >>>>>> } >>>>>> EndpointResolver resolver = >>>>>> client.createResolverForService(destinationQname); >>>>>> client.send(resolver, null, null, new >>>>>> StringSource(writer.toString())); >>>>>> >>>>>> I'm not missing anything in the above code am I? >>>>>> Thanks, >>>>>> Ryan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> > >
