Hi,
This should be a threadPool issue.
To avoid the hanging thread, you can set queueSize=0 and maximumPoolSize
= -1 for your servicemix-camel component.
With this configuration, for new incoming request, a new thread will be
created, so that avoid the deadlock.
This may cause a problem(too many threads created) when you have too
much concurrent request,
but if the endpoint you see have the thread problem play the role as
provider, I think one solution is that you can also add thread pool on
consumer side so that you can control the concurrent invocation number
arrive the provider to avoid the resource run out problem.
For example, the work flow looks like
external client ===> cxf bc consumer ===> your SE (servicemix-camel in
your case)
Although you can't control how many concurrent invocation comes to cxf
bc consumer, but you can configure the cxf bc consumer thread pool size
(by configure the underlying jetty)to determine how many invocation can
dispatch to your SE at the same time.
This is just a example, hopefully you can get roughly idea how things
can work.
Ideally, the concurrent request should be less than the corePoolSize for
component when you use sendSync (which potentially have thread deadlock
issue when corePoolSize is run out)
Another way to avoid the deadlock is use send instead.
[1] get more details about threadPool configuration
[1]http://servicemix.apache.org/thread-pools.html
Freeman
navigator09 wrote:
Hi,
I think there is a deadlock situation between two(6 and 16) threads and
the two threads will simply wait till the they time out. I have pasted the
logs produced before servicemix stops all activity
19:51:23,171 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
SecuredBroker | mix.jbi.security.SecuredBroker 66 | send
exchange with secure broker
19:51:23,187 | INFO | pool-flow.seda.servicemix-camel-thread-6 |
jobsubmissionoutput | .apache.camel.processor.Logger 88 |
Exchange[BodyType:javax.xml.transform.dom.DOMSource, Body:<submitJobResponse
xmlns="http://www.icenigrid.org/service/gridsam";
xmlns:ns2="http://schemas.ggf.org/jsdl/2005/11/jsdl";><JobIdentifier><ID>{"WorkingDirectory":"uploadfiles","ServerIP":"10.20.3.97","Password":"icmconct","UserName":"icmconct","JobDetails":"940:0"}</ID></JobIdentifier></submitJobResponse>]
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
DeliveryChannelImpl | .messaging.DeliveryChannelImpl 458 | SendSync
ID:10.20.3.92-11e88445927-2:33 in DeliveryChannel{servicemix-camel}
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
SecuredBroker | mix.jbi.security.SecuredBroker 72 | service
name :{http://foo.bar.org}EventProcessor
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
SecuredBroker | mix.jbi.security.SecuredBroker 66 | send
exchange with secure broker
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
SecuredBroker | mix.jbi.security.SecuredBroker 73 | operation
name :{http://www.icenigrid.org/service/gridsam}submitJob
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
SecuredBroker | mix.jbi.security.SecuredBroker 72 | service
name :{http://foo.bar.org}EventProcessor
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
SecuredBroker | mix.jbi.security.SecuredBroker 73 | operation
name :{http://www.icenigrid.org/service/gridsam}submitJob
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | SedaFlow
| emix.jbi.nmr.flow.AbstractFlow 118 | Called Flow send
19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
DeliveryChannelImpl | .messaging.DeliveryChannelImpl 693 | Waiting for
exchange ID:10.20.3.92-11e88445927-2:32 (19b9669) to be answered in
DeliveryChannel{servicemix-camel} from sendSync
19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | SedaFlow
| emix.jbi.nmr.flow.AbstractFlow 118 | Called Flow send
19:51:23,375 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
DeliveryChannelImpl | .messaging.DeliveryChannelImpl 693 | Waiting for
exchange ID:10.20.3.92-11e88445927-2:33 (1afe611) to be answered in
DeliveryChannel{servicemix-camel} from sendSync
How do I solve the problem?
Regards,
Ganesh
navigator09 wrote:
Hi,
I am load testing the following configuration
from("jbi:endpoint:http://foo.bar.org/Pipeline/Endpoint?mep=in-out";).
to("jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out";).
convertBodyTo(DOMSource.class).
to("log:jobsubmissionoutput").
to("jbi:endpoint:http://foo.bar.org/EventProcessor/EventProcessorendpoint";).
The entire flow works well with 7 concurrent threads. But when I try with
more threads , the flow works fine till
jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out
end point. The response from this end point is not forwrded to the event
processor end point and I get the following message
19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-3 |
SecuredBroker | mix.jbi.security.SecuredBroker 72 | service
name :{http://foo.bar.org}EventProcessor
19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
DeliveryChannelImpl | .messaging.DeliveryChannelImpl 693 | Waiting
for exchange ID:10.20.3.92-11e881736ec-2:19 (de1520) to be answered in
DeliveryChannel{servicemix-camel} from sendSync
Then after waiting for around 10 minutes the the thread expires ( as
configured in component.properties). Why is the event processor not being
processed properly for 10 and above concorrent threads.
Regards,
Ganesh
