If the endpoint you see have the thread problem play the role as
provider, I think one solution is that you can also add thread pool on
consumer side so that you can control the concurrent invocation number
arrive the provider to avoid the resource run out problem as you said.
For example, the work flow looks like
external client ===> cxf bc consumer ===> your SE
Although you can't control how many concurrent invocation comes to cxf
bc consumer, but you can configure the cxf bc consumer thread pool size
(by configure the underlying jetty)to determine how many invocation can
dispatch to your SE at the same time.
This is just a example, hopefully you can get roughly idea how things
can work.
Freeman
Fabian Crabus wrote:
This may be a solution to avoid hanging threads, but during high load we'll
quickly run out of database
connections. So we've minimized the maximumPoolSize in the latest tests.
And getting rid of the queue can't be the way either...then we could just
use the ST-flow, right?
On Wed, Nov 26, 2008 at 13:15, Freeman Fang <[EMAIL PROTECTED]> wrote:
Hi,
How about change queueSize=0, so that when one task need be executed(this
task may notify your other wait thread), it will not go in the queue but a
new thread will be created since your maximumPoolSize is -1.
Freeman
Tomasz Prus wrote:
Hi Fabian,
Do you have any idea because we have similar problem?
Tomasz Prus
2008/11/21 Fabian Crabus <[EMAIL PROTECTED]>:
Hi,
we see a very strange behaviour in our ServiceMix configuration. We
process
a few thousand messages
and then after some time ServiceMix comes to a complete halt.
The thread dump shows something like this:
"pool-component.resequencer-rulebaseRouter-thread-1" prio=5 tid=0x55
nid=0x0
timed_waiting []
java.lang.Thread$State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.
java:1879)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:286)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.accept(DeliveryChannelImpl.java:270)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.pollDeliveryChannel(AsyncBaseLifeCycle.java:314)
at
org.apache.servicemix.common.AsyncBaseLifeCycle$1.run(AsyncBaseLifeCycle.java:300)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
"pool-flow.seda.resequencer-rulebaseRouter-thread-9" prio=5 tid=0x9c
nid=0x0
waiting []
java.lang.Thread$State: WAITING
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:
1841)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:170)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Nothing is being processed. All threads are waiting. We've changed the
thread pool configuration to these values:
<bean class="org.apache.servicemix.executors.impl.ExecutorConfig">
<property name="corePoolSize" value="30"/>
<property name="" value="-1"/>
<property name="queueSize" value="512"/>
</bean>
As far as I understand there are several ways to get to these errors:
- not sending DONE or ERROR
- having to few threads
- deadlock caused by circular sendSync calls
We're examining our code if one of these could be the culprit.
But are there other situation where this could happen (even though there
are
"limitless" potential threads available -> maximumPoolSize=-1)?
And another thing I don't understand in the thread dump:
we only have one thread
"pool-component.resequencer-rulebaseRouter-thread-1",
i.e. no other pool-component threads. Is that normal?
Why is it different to the "pool-seda" threads?
Thanks for any pointers,
Fabian
