Hi Freeman, I get your point. The number of threads created for the servicemix-camel was insufficient to cater to the requests for the threads created by the consumer side. So by adding qeueSize=0 and maximumPoolSize = -1( creating as many concurrent threads as requests) to the servicemix-camel component, the bottleneck is solved.
Regards, Ganesh Freeman Fang wrote: > > Hi, > > This should be a threadPool issue. > To avoid the hanging thread, you can set queueSize=0 and maximumPoolSize > = -1 for your servicemix-camel component. > With this configuration, for new incoming request, a new thread will be > created, so that avoid the deadlock. > This may cause a problem(too many threads created) when you have too > much concurrent request, > but if the endpoint you see have the thread problem play the role as > provider, I think one solution is that you can also add thread pool on > consumer side so that you can control the concurrent invocation number > arrive the provider to avoid the resource run out problem. > > For example, the work flow looks like > external client ===> cxf bc consumer ===> your SE (servicemix-camel in > your case) > Although you can't control how many concurrent invocation comes to cxf > bc consumer, but you can configure the cxf bc consumer thread pool size > (by configure the underlying jetty)to determine how many invocation can > dispatch to your SE at the same time. > This is just a example, hopefully you can get roughly idea how things > can work. > Ideally, the concurrent request should be less than the corePoolSize for > component when you use sendSync (which potentially have thread deadlock > issue when corePoolSize is run out) > > Another way to avoid the deadlock is use send instead. > > [1] get more details about threadPool configuration > [1]http://servicemix.apache.org/thread-pools.html > > Freeman > > navigator09 wrote: >> Hi, >> >> I think there is a deadlock situation between two(6 and 16) threads and >> the two threads will simply wait till the they time out. I have pasted >> the >> logs produced before servicemix stops all activity >> >> 19:51:23,171 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | >> SecuredBroker | mix.jbi.security.SecuredBroker 66 | send >> exchange with secure broker >> 19:51:23,187 | INFO | pool-flow.seda.servicemix-camel-thread-6 | >> jobsubmissionoutput | .apache.camel.processor.Logger 88 | >> Exchange[BodyType:javax.xml.transform.dom.DOMSource, >> Body:<submitJobResponse >> xmlns="http://www.icenigrid.org/service/gridsam"; >> xmlns:ns2="http://schemas.ggf.org/jsdl/2005/11/jsdl";><JobIdentifier><ID>{"WorkingDirectory":"uploadfiles","ServerIP":"10.20.3.97","Password":"icmconct","UserName":"icmconct","JobDetails":"940:0"}</ID></JobIdentifier></submitJobResponse>] >> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | >> DeliveryChannelImpl | .messaging.DeliveryChannelImpl 458 | SendSync >> ID:10.20.3.92-11e88445927-2:33 in DeliveryChannel{servicemix-camel} >> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | >> SecuredBroker | mix.jbi.security.SecuredBroker 72 | service >> name :{http://foo.bar.org}EventProcessor >> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | >> SecuredBroker | mix.jbi.security.SecuredBroker 66 | send >> exchange with secure broker >> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | >> SecuredBroker | mix.jbi.security.SecuredBroker 73 | >> operation >> name :{http://www.icenigrid.org/service/gridsam}submitJob >> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | >> SecuredBroker | mix.jbi.security.SecuredBroker 72 | service >> name :{http://foo.bar.org}EventProcessor >> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | >> SecuredBroker | mix.jbi.security.SecuredBroker 73 | >> operation >> name :{http://www.icenigrid.org/service/gridsam}submitJob >> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | >> SedaFlow >> | emix.jbi.nmr.flow.AbstractFlow 118 | Called Flow send >> 19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | >> DeliveryChannelImpl | .messaging.DeliveryChannelImpl 693 | Waiting >> for >> exchange ID:10.20.3.92-11e88445927-2:32 (19b9669) to be answered in >> DeliveryChannel{servicemix-camel} from sendSync >> 19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | >> SedaFlow >> | emix.jbi.nmr.flow.AbstractFlow 118 | Called Flow send >> 19:51:23,375 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | >> DeliveryChannelImpl | .messaging.DeliveryChannelImpl 693 | Waiting >> for >> exchange ID:10.20.3.92-11e88445927-2:33 (1afe611) to be answered in >> DeliveryChannel{servicemix-camel} from sendSync >> >> How do I solve the problem? >> >> Regards, >> Ganesh >> >> navigator09 wrote: >> >>> Hi, >>> >>> I am load testing the following configuration >>> >>> from("jbi:endpoint:http://foo.bar.org/Pipeline/Endpoint?mep=in-out";). >>> >>> to("jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out";). >>> convertBodyTo(DOMSource.class). >>> to("log:jobsubmissionoutput"). >>> >>> to("jbi:endpoint:http://foo.bar.org/EventProcessor/EventProcessorendpoint";). >>> >>> The entire flow works well with 7 concurrent threads. But when I try >>> with >>> more threads , the flow works fine till >>> jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out >>> end point. The response from this end point is not forwrded to the event >>> processor end point and I get the following message >>> >>> 19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-3 | >>> SecuredBroker | mix.jbi.security.SecuredBroker 72 | service >>> name :{http://foo.bar.org}EventProcessor >>> 19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | >>> DeliveryChannelImpl | .messaging.DeliveryChannelImpl 693 | Waiting >>> for exchange ID:10.20.3.92-11e881736ec-2:19 (de1520) to be answered in >>> DeliveryChannel{servicemix-camel} from sendSync >>> >>> Then after waiting for around 10 minutes the the thread expires ( as >>> configured in component.properties). Why is the event processor not >>> being >>> processed properly for 10 and above concorrent threads. >>> >>> Regards, >>> Ganesh >>> >>> >> >> > > > -- View this message in context: http://www.nabble.com/Load-testing-problems-with-servicemix-camel-component-tp21217362p21227353.html Sent from the ServiceMix - User mailing list archive at Nabble.com.
