Hi Gert,
I actually decided to do that rather than go the the stack trace way, since
it seemed like it still going to be like searching for a needle in a
haystack. Anyhow, I tried to reproduce this problem using basically a
simple jms bridge service archive where I setup my JMS service units the
same as my full blown project. When I did that, I was able to pump 2000
messages across them without any problems at a rate of one message per 50
ms.
So, basically I've shot my whole theory out of the water that it has
anything to do with ActiveMQ refusing messages. I guess I'll need to add a
servicemix-bean component to forward requests to the jms provider rather
than from the jms consumer to the jms provider.
I'll keep working on it until I get it because I really need to figure out
this problem since it's hindering several projects now. No idea why all of
a sudden the problem became so severe.
On Sat, Aug 23, 2008 at 9:47 AM, Gert Vanthienen
<[EMAIL PROTECTED]>wrote:
> Ryan,
>
> Do you think you could build a simple SA to reproduce the issue (e.g. with
> a quartz SU or a Camel time that sends a lot of exchanges to the topic)? If
> so, could you raise a JIRA and attach the SA so we can look into this?
>
>
> Regards,
>
> Gert
>
> Ryan Moquin wrote:
>
>> I changed my broker in my activemq.xml in servicemix like this:
>>
>> <!-- ActiveMQ JMS Broker configuration -->
>> <amq:broker id="broker" depends-on="jmxServer" useShutdownHook="false"
>> persistent="false">
>>
>> and then started it up. It went to 1444 and then stopped processing
>> anything. So my guess is that that doesn't help either. I think some of
>> my
>> other services are able to keep sender so it seems like the producer flow
>> control is kicking in rather than my messages being expired like I want.
>>
>> On Fri, Aug 22, 2008 at 7:43 AM, Gert Vanthienen
>> <[EMAIL PROTECTED]>wrote:
>>
>>
>>
>>> Ryan,
>>>
>>> Odd... Just to be sure we are not looking at something completely
>>> different
>>> here, could you try to disable persistence on the broker alltogether to
>>> ensure that this fixes the problem? From the stacktraces, it looks like
>>> it
>>> is the SEDA flow itself that is hanging here (waiting for thread to
>>> become
>>> available after a sendSync to somewhere perhaps?)
>>>
>>> Regards,
>>>
>>> Gert
>>>
>>>
>>> Ryan Moquin wrote:
>>>
>>>
>>>
>>>> Just a little bit more. So everything I find on this issue, seems to
>>>> indicate I guess that since the JMS delivery mode is persistent by
>>>> default,
>>>> you should turn off persistence if you don't want a producer to block.
>>>> So
>>>> since I don't want my service to block, nor do I want the messages
>>>> persistent, I configured my jms provider like this:
>>>>
>>>> <jms:provider service="not:notification-jms-service"
>>>> endpoint="notification"
>>>> pubSubDomain="true"
>>>> destinationName="topic.notication"
>>>> connectionFactory="#connectionFactory"
>>>> deliveryMode="1"
>>>> timeToLive="1000"/>
>>>>
>>>> As far as I can tell (since it's not documented what the values of
>>>> deliveryMode are), this should keep my producer from blocking, but it
>>>> still
>>>> does after it hits a little over a 1000 messages. Is there a way to
>>>> turn
>>>> off persistence in my producers that work? I have not found a way to
>>>> make
>>>> this happen.
>>>>
>>>> On Tue, Aug 19, 2008 at 7:54 PM, Ryan Moquin <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Maybe this is the better way to ask this as I try to rationalize this
>>>>> behavior in my head:
>>>>>
>>>>> If you have a service unit that sends messages to a topic created by a
>>>>> servicemix-jms provider endpoint that has no consumers, shouldn't every
>>>>> message sent to it, be discarded? I'm not really sure I actually know
>>>>> what
>>>>> the problem is, since all my configuration attempts to fix it have
>>>>> failed.
>>>>> All I want at this point is any way possible to prevent the system from
>>>>> deadlocking. I'd be happy going back to exceptions being thrown all
>>>>> the
>>>>> time instead. At one point with Servicemix 3.2.2, we've processed lots
>>>>> and
>>>>> lots of messages (eventually Servicemix ran out of memory and crashed,
>>>>> no
>>>>> clue as to why that happened consistently every 2 hours, but that's
>>>>> another
>>>>> issue, I'm guessing because ActiveMQ never discards any messages). We
>>>>> haven't changed any of the servicemix configurations since then, so I'm
>>>>> not
>>>>> sure exactly why we can only process a small amount before Servicemix
>>>>> will
>>>>> hang from ActiveMQ blocking.
>>>>>
>>>>> I'm very puzzled at this point.
>>>>>
>>>>>
>>>>> On Tue, Aug 19, 2008 at 10:33 AM, Ryan Moquin <[EMAIL PROTECTED]
>>>>>
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> So, due to my service units hanging under certain conditions when I
>>>>>> try
>>>>>> to
>>>>>> send a JMS message using an injected DeliveryChannel, I switched to
>>>>>> using a
>>>>>> ServicemixClient I look up out of the servicemix JNDI context. This
>>>>>> is
>>>>>> done
>>>>>> by several of my deployed services that are doing various pieces of
>>>>>> work.
>>>>>> I'm now noticing that while switching to a separate ServicemixClient
>>>>>> has
>>>>>> fixed some of my issues, I'm now seeing certain services that send
>>>>>> messages
>>>>>> to a JMS service unit to send out of the syste, are also now
>>>>>> deadlocking
>>>>>> around the same point and therefore my service halts. I did a thread
>>>>>> dump
>>>>>> to verify this and I see A LOT of threads that are waiting on a
>>>>>> specific
>>>>>> lock with id: 0x1abeba80. You can see one of these below:
>>>>>>
>>>>>> "pool-flow.seda.servicemix-jms-thread-1157" prio=6 tid=0x3cee2400
>>>>>> nid=0x1a00 waiting on condition [0
>>>>>> x3869f000..0x3869fa98]
>>>>>> java.lang.Thread.State: WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for <0x1abeba80> (a
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer
>>>>>> $ConditionObject)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
>>>>>> dSynchronizer.java:1925)
>>>>>> at
>>>>>>
>>>>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
>>>>>> l.java:663)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>
>>>>>> My service itself is also locking on the same id, while trying to send
>>>>>> a
>>>>>> message to the JMS consumer (I think is the right term):
>>>>>>
>>>>>> "QuartzScheduler_Worker-7" prio=6 tid=0x3a2f1400 nid=0x1050 waiting on
>>>>>> condition [0x37e2e000..0x37e2
>>>>>> fc18]
>>>>>> java.lang.Thread.State: WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for <0x1abeba80> (a
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer
>>>>>> $ConditionObject)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
>>>>>> dSynchronizer.java:1925)
>>>>>> at
>>>>>>
>>>>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
>>>>>> l.java:663)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
>>>>>> utor.java:1737)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
>>>>>> )
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
>>>>>> 95)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
>>>>>> )
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jms.multiplexing.MultiplexingProviderProcessor.process(Multiplexing
>>>>>> ProviderProcessor.java:143)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:538)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:4
>>>>>> 90)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.common.BaseLifeCycle.onMessageExchange(BaseLifeCycle.java:46)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
>>>>>> l.java:610)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
>>>>>> utor.java:1737)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
>>>>>> )
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
>>>>>> 95)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
>>>>>> )
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:14
>>>>>> 7)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:21
>>>>>> 1)
>>>>>>
>>>>>> So my question is, is activemq freezing up for some reason? All my
>>>>>> problems I have, seem to somehow stem to activemq in various ways,
>>>>>> usually
>>>>>> all my freeze ups seem to be attributed to ActiveMQ freezing up for
>>>>>> one
>>>>>> reason or another. If the above case is that ActiveMQ is freezing up
>>>>>> (it
>>>>>> happens over a period of time, I thought I read that ActiveMQ will
>>>>>> block
>>>>>> when a topic or Queue fills up, which makes no sense to me in regards
>>>>>> to
>>>>>> a
>>>>>> topic, since I thought that a topic doesn't hold messages unless they
>>>>>> are
>>>>>> marked durable), are there settings I can use to keep this from
>>>>>> happening?
>>>>>> There is no way I can ever trust this sort of system in production
>>>>>> unless I
>>>>>> can be sure that Servicemix/ActiveMQ isn't going to lock up on me
>>>>>> after
>>>>>> a
>>>>>> couple hours.
>>>>>>
>>>>>> In case you want to see how I'm using the ServicemixClient, I'm not
>>>>>> doing
>>>>>> anything overly special, but here it is:
>>>>>>
>>>>>> if (client == null) {
>>>>>> ClientFactory factory = (ClientFactory) new
>>>>>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
>>>>>> client = factory.createClient();
>>>>>> }
>>>>>>
>>>>>> if (destinationQname == null) {
>>>>>> destinationQname = new QName(getDestinationNamespace(),
>>>>>> getDestinationService());
>>>>>> }
>>>>>> EndpointResolver resolver =
>>>>>> client.createResolverForService(destinationQname);
>>>>>> client.send(resolver, null, null, new
>>>>>> StringSource(writer.toString()));
>>>>>>
>>>>>> I'm not missing anything in the above code am I?
>>>>>> Thanks,
>>>>>> Ryan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
