Ryan,
Do you think you could build a simple SA to reproduce the issue (e.g.
with a quartz SU or a Camel time that sends a lot of exchanges to the
topic)? If so, could you raise a JIRA and attach the SA so we can look
into this?
Regards,
Gert
Ryan Moquin wrote:
I changed my broker in my activemq.xml in servicemix like this:
<!-- ActiveMQ JMS Broker configuration -->
<amq:broker id="broker" depends-on="jmxServer" useShutdownHook="false"
persistent="false">
and then started it up. It went to 1444 and then stopped processing
anything. So my guess is that that doesn't help either. I think some of my
other services are able to keep sender so it seems like the producer flow
control is kicking in rather than my messages being expired like I want.
On Fri, Aug 22, 2008 at 7:43 AM, Gert Vanthienen
<[EMAIL PROTECTED]>wrote:
Ryan,
Odd... Just to be sure we are not looking at something completely different
here, could you try to disable persistence on the broker alltogether to
ensure that this fixes the problem? From the stacktraces, it looks like it
is the SEDA flow itself that is hanging here (waiting for thread to become
available after a sendSync to somewhere perhaps?)
Regards,
Gert
Ryan Moquin wrote:
Just a little bit more. So everything I find on this issue, seems to
indicate I guess that since the JMS delivery mode is persistent by
default,
you should turn off persistence if you don't want a producer to block. So
since I don't want my service to block, nor do I want the messages
persistent, I configured my jms provider like this:
<jms:provider service="not:notification-jms-service"
endpoint="notification"
pubSubDomain="true"
destinationName="topic.notication"
connectionFactory="#connectionFactory"
deliveryMode="1"
timeToLive="1000"/>
As far as I can tell (since it's not documented what the values of
deliveryMode are), this should keep my producer from blocking, but it
still
does after it hits a little over a 1000 messages. Is there a way to turn
off persistence in my producers that work? I have not found a way to make
this happen.
On Tue, Aug 19, 2008 at 7:54 PM, Ryan Moquin <[EMAIL PROTECTED]>
wrote:
Maybe this is the better way to ask this as I try to rationalize this
behavior in my head:
If you have a service unit that sends messages to a topic created by a
servicemix-jms provider endpoint that has no consumers, shouldn't every
message sent to it, be discarded? I'm not really sure I actually know
what
the problem is, since all my configuration attempts to fix it have
failed.
All I want at this point is any way possible to prevent the system from
deadlocking. I'd be happy going back to exceptions being thrown all the
time instead. At one point with Servicemix 3.2.2, we've processed lots
and
lots of messages (eventually Servicemix ran out of memory and crashed, no
clue as to why that happened consistently every 2 hours, but that's
another
issue, I'm guessing because ActiveMQ never discards any messages). We
haven't changed any of the servicemix configurations since then, so I'm
not
sure exactly why we can only process a small amount before Servicemix
will
hang from ActiveMQ blocking.
I'm very puzzled at this point.
On Tue, Aug 19, 2008 at 10:33 AM, Ryan Moquin <[EMAIL PROTECTED]
wrote:
Hello,
So, due to my service units hanging under certain conditions when I try
to
send a JMS message using an injected DeliveryChannel, I switched to
using a
ServicemixClient I look up out of the servicemix JNDI context. This is
done
by several of my deployed services that are doing various pieces of
work.
I'm now noticing that while switching to a separate ServicemixClient has
fixed some of my issues, I'm now seeing certain services that send
messages
to a JMS service unit to send out of the syste, are also now deadlocking
around the same point and therefore my service halts. I did a thread
dump
to verify this and I see A LOT of threads that are waiting on a specific
lock with id: 0x1abeba80. You can see one of these below:
"pool-flow.seda.servicemix-jms-thread-1157" prio=6 tid=0x3cee2400
nid=0x1a00 waiting on condition [0
x3869f000..0x3869fa98]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1abeba80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
dSynchronizer.java:1925)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
My service itself is also locking on the same id, while trying to send a
message to the JMS consumer (I think is the right term):
"QuartzScheduler_Worker-7" prio=6 tid=0x3a2f1400 nid=0x1050 waiting on
condition [0x37e2e000..0x37e2
fc18]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1abeba80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
dSynchronizer.java:1925)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
utor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
95)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
)
at
org.apache.servicemix.jms.multiplexing.MultiplexingProviderProcessor.process(Multiplexing
ProviderProcessor.java:143)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:538)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:4
90)
at
org.apache.servicemix.common.BaseLifeCycle.onMessageExchange(BaseLifeCycle.java:46)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
l.java:610)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
utor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
95)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:14
7)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:21
1)
So my question is, is activemq freezing up for some reason? All my
problems I have, seem to somehow stem to activemq in various ways,
usually
all my freeze ups seem to be attributed to ActiveMQ freezing up for one
reason or another. If the above case is that ActiveMQ is freezing up
(it
happens over a period of time, I thought I read that ActiveMQ will block
when a topic or Queue fills up, which makes no sense to me in regards to
a
topic, since I thought that a topic doesn't hold messages unless they
are
marked durable), are there settings I can use to keep this from
happening?
There is no way I can ever trust this sort of system in production
unless I
can be sure that Servicemix/ActiveMQ isn't going to lock up on me after
a
couple hours.
In case you want to see how I'm using the ServicemixClient, I'm not
doing
anything overly special, but here it is:
if (client == null) {
ClientFactory factory = (ClientFactory) new
InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
client = factory.createClient();
}
if (destinationQname == null) {
destinationQname = new QName(getDestinationNamespace(),
getDestinationService());
}
EndpointResolver resolver =
client.createResolverForService(destinationQname);
client.send(resolver, null, null, new
StringSource(writer.toString()));
I'm not missing anything in the above code am I?
Thanks,
Ryan
