Ryan,
Odd... Just to be sure we are not looking at something completely
different here, could you try to disable persistence on the broker
alltogether to ensure that this fixes the problem?
From the stacktraces, it looks like it is the SEDA flow itself that is
hanging here (waiting for thread to become available after a sendSync to
somewhere perhaps?)
Regards,
Gert
Ryan Moquin wrote:
Just a little bit more. So everything I find on this issue, seems to
indicate I guess that since the JMS delivery mode is persistent by default,
you should turn off persistence if you don't want a producer to block. So
since I don't want my service to block, nor do I want the messages
persistent, I configured my jms provider like this:
<jms:provider service="not:notification-jms-service"
endpoint="notification"
pubSubDomain="true"
destinationName="topic.notication"
connectionFactory="#connectionFactory"
deliveryMode="1"
timeToLive="1000"/>
As far as I can tell (since it's not documented what the values of
deliveryMode are), this should keep my producer from blocking, but it still
does after it hits a little over a 1000 messages. Is there a way to turn
off persistence in my producers that work? I have not found a way to make
this happen.
On Tue, Aug 19, 2008 at 7:54 PM, Ryan Moquin <[EMAIL PROTECTED]> wrote:
Maybe this is the better way to ask this as I try to rationalize this
behavior in my head:
If you have a service unit that sends messages to a topic created by a
servicemix-jms provider endpoint that has no consumers, shouldn't every
message sent to it, be discarded? I'm not really sure I actually know what
the problem is, since all my configuration attempts to fix it have failed.
All I want at this point is any way possible to prevent the system from
deadlocking. I'd be happy going back to exceptions being thrown all the
time instead. At one point with Servicemix 3.2.2, we've processed lots and
lots of messages (eventually Servicemix ran out of memory and crashed, no
clue as to why that happened consistently every 2 hours, but that's another
issue, I'm guessing because ActiveMQ never discards any messages). We
haven't changed any of the servicemix configurations since then, so I'm not
sure exactly why we can only process a small amount before Servicemix will
hang from ActiveMQ blocking.
I'm very puzzled at this point.
On Tue, Aug 19, 2008 at 10:33 AM, Ryan Moquin <[EMAIL PROTECTED]>wrote:
Hello,
So, due to my service units hanging under certain conditions when I try to
send a JMS message using an injected DeliveryChannel, I switched to using a
ServicemixClient I look up out of the servicemix JNDI context. This is done
by several of my deployed services that are doing various pieces of work.
I'm now noticing that while switching to a separate ServicemixClient has
fixed some of my issues, I'm now seeing certain services that send messages
to a JMS service unit to send out of the syste, are also now deadlocking
around the same point and therefore my service halts. I did a thread dump
to verify this and I see A LOT of threads that are waiting on a specific
lock with id: 0x1abeba80. You can see one of these below:
"pool-flow.seda.servicemix-jms-thread-1157" prio=6 tid=0x3cee2400
nid=0x1a00 waiting on condition [0
x3869f000..0x3869fa98]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1abeba80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
dSynchronizer.java:1925)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
My service itself is also locking on the same id, while trying to send a
message to the JMS consumer (I think is the right term):
"QuartzScheduler_Worker-7" prio=6 tid=0x3a2f1400 nid=0x1050 waiting on
condition [0x37e2e000..0x37e2
fc18]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1abeba80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
dSynchronizer.java:1925)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
utor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
95)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
)
at
org.apache.servicemix.jms.multiplexing.MultiplexingProviderProcessor.process(Multiplexing
ProviderProcessor.java:143)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:538)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:4
90)
at
org.apache.servicemix.common.BaseLifeCycle.onMessageExchange(BaseLifeCycle.java:46)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:610)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
utor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
95)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:14
7)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:21
1)
So my question is, is activemq freezing up for some reason? All my
problems I have, seem to somehow stem to activemq in various ways, usually
all my freeze ups seem to be attributed to ActiveMQ freezing up for one
reason or another. If the above case is that ActiveMQ is freezing up (it
happens over a period of time, I thought I read that ActiveMQ will block
when a topic or Queue fills up, which makes no sense to me in regards to a
topic, since I thought that a topic doesn't hold messages unless they are
marked durable), are there settings I can use to keep this from happening?
There is no way I can ever trust this sort of system in production unless I
can be sure that Servicemix/ActiveMQ isn't going to lock up on me after a
couple hours.
In case you want to see how I'm using the ServicemixClient, I'm not doing
anything overly special, but here it is:
if (client == null) {
ClientFactory factory = (ClientFactory) new
InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
client = factory.createClient();
}
if (destinationQname == null) {
destinationQname = new QName(getDestinationNamespace(),
getDestinationService());
}
EndpointResolver resolver =
client.createResolverForService(destinationQname);
client.send(resolver, null, null, new
StringSource(writer.toString()));
I'm not missing anything in the above code am I?
Thanks,
Ryan
