Hello,
So, due to my service units hanging under certain conditions when I try to
send a JMS message using an injected DeliveryChannel, I switched to using a
ServicemixClient I look up out of the servicemix JNDI context. This is done
by several of my deployed services that are doing various pieces of work.
I'm now noticing that while switching to a separate ServicemixClient has
fixed some of my issues, I'm now seeing certain services that send messages
to a JMS service unit to send out of the syste, are also now deadlocking
around the same point and therefore my service halts. I did a thread dump
to verify this and I see A LOT of threads that are waiting on a specific
lock with id: 0x1abeba80. You can see one of these below:
"pool-flow.seda.servicemix-jms-thread-1157" prio=6 tid=0x3cee2400 nid=0x1a00
waiting on condition [0
x3869f000..0x3869fa98]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1abeba80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
dSynchronizer.java:1925)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
My service itself is also locking on the same id, while trying to send a
message to the JMS consumer (I think is the right term):
"QuartzScheduler_Worker-7" prio=6 tid=0x3a2f1400 nid=0x1050 waiting on
condition [0x37e2e000..0x37e2
fc18]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1abeba80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
dSynchronizer.java:1925)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
utor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
95)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
)
at
org.apache.servicemix.jms.multiplexing.MultiplexingProviderProcessor.process(Multiplexing
ProviderProcessor.java:143)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:538)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:4
90)
at
org.apache.servicemix.common.BaseLifeCycle.onMessageExchange(BaseLifeCycle.java:46)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:610)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
utor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
95)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:14
7)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:21
1)
So my question is, is activemq freezing up for some reason? All my problems
I have, seem to somehow stem to activemq in various ways, usually all my
freeze ups seem to be attributed to ActiveMQ freezing up for one reason or
another. If the above case is that ActiveMQ is freezing up (it happens over
a period of time, I thought I read that ActiveMQ will block when a topic or
Queue fills up, which makes no sense to me in regards to a topic, since I
thought that a topic doesn't hold messages unless they are marked durable),
are there settings I can use to keep this from happening? There is no way I
can ever trust this sort of system in production unless I can be sure that
Servicemix/ActiveMQ isn't going to lock up on me after a couple hours.
In case you want to see how I'm using the ServicemixClient, I'm not doing
anything overly special, but here it is:
if (client == null) {
ClientFactory factory = (ClientFactory) new
InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
client = factory.createClient();
}
if (destinationQname == null) {
destinationQname = new QName(getDestinationNamespace(),
getDestinationService());
}
EndpointResolver resolver =
client.createResolverForService(destinationQname);
client.send(resolver, null, null, new
StringSource(writer.toString()));
I'm not missing anything in the above code am I?
Thanks,
Ryan
