Hi,
we see a very strange behaviour in our ServiceMix configuration. We process
a few thousand messages
and then after some time ServiceMix comes to a complete halt.
The thread dump shows something like this:
"pool-component.resequencer-rulebaseRouter-thread-1" prio=5 tid=0x55 nid=0x0
timed_waiting []
java.lang.Thread$State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.
java:1879)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:286)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.accept(DeliveryChannelImpl.java:270)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.pollDeliveryChannel(AsyncBaseLifeCycle.java:314)
at
org.apache.servicemix.common.AsyncBaseLifeCycle$1.run(AsyncBaseLifeCycle.java:300)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
"pool-flow.seda.resequencer-rulebaseRouter-thread-9" prio=5 tid=0x9c nid=0x0
waiting []
java.lang.Thread$State: WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:
1841)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:170)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Nothing is being processed. All threads are waiting. We've changed the
thread pool configuration to these values:
<bean class="org.apache.servicemix.executors.impl.ExecutorConfig">
<property name="corePoolSize" value="30"/>
<property name="maximumPoolSize" value="-1"/>
<property name="queueSize" value="512"/>
</bean>
As far as I understand there are several ways to get to these errors:
- not sending DONE or ERROR
- having to few threads
- deadlock caused by circular sendSync calls
We're examining our code if one of these could be the culprit.
But are there other situation where this could happen (even though there are
"limitless" potential threads available -> maximumPoolSize=-1)?
And another thing I don't understand in the thread dump:
we only have one thread "pool-component.resequencer-rulebaseRouter-thread-1",
i.e. no other pool-component threads. Is that normal?
Why is it different to the "pool-seda" threads?
Thanks for any pointers,
Fabian
