We have committed this fix and now ESB tests running fine.
On Mon, Nov 18, 2013 at 10:54 AM, Krishantha Samaraweera < [email protected]> wrote: > Thanks Miyuru, I will check the fix too.. > > > > > On Fri, Nov 15, 2013 at 12:49 PM, Miyuru Wanninayaka <[email protected]>wrote: > >> >> >> >> On Fri, Nov 15, 2013 at 12:29 PM, Miyuru Wanninayaka <[email protected]>wrote: >> >>> Seems once proxy service deploy/undploy lock in axis2 config instead of >>> synapse config, no more deadlocks :) >>> >>> I have to >>> >>> - private AxisConfiguration axisConfiguration = null; >>> + private AxisConfiguration axisConfiguration = new >>> AxisConfiguration(); >>> >>> because in MultiXMLConfigurationBuilderTest, it creates a synapse config >>> without axisConfig which will cause NPE in synchronization block. >>> >>> I'm currently running tests with these changes. >>> >>> >>> On Fri, Nov 15, 2013 at 10:37 AM, Miyuru Wanninayaka <[email protected]>wrote: >>> >>>> May be for proxy service undeployment, synapse can lock on axis2 config >>>> instead of synapse config, which will avoid deadlocking. >>>> >>>> >>>> On Fri, Nov 15, 2013 at 10:22 AM, Miyuru Wanninayaka >>>> <[email protected]>wrote: >>>> >>>>> I have observed this few times even without patch0002. >>>>> >>>>> When analyzed deadloak flow it seems >>>>> hot deployment of main, fault sequences and deleting ESB artifacts >>>>> from admin services triggers in parallel. >>>>> >>>>> in hot deployment, axis2 thread first get a lock of axis2 >>>>> configuration and then calls synapse deployer.undeploy which will lock on >>>>> synapse configuration. >>>>> during admin service call, it will directly call synapse to undeploy >>>>> artifacts which will first lock on synapse config and in case of proxy, it >>>>> will cause related axis2 service to undeploy and try to acquire lock of >>>>> axis2 config which will become a deadlock. >>>>> >>>>> I have modified abstract ESBIntegrationTest.cleanup() method to wait >>>>> few seconds to let main and fault sequence to hot update. Seems deadlock >>>>> does not happens after that. >>>>> >>>>> >>>>> On Thu, Nov 14, 2013 at 10:17 PM, Miyuru Wanninayaka >>>>> <[email protected]>wrote: >>>>> >>>>>> Seems patch0002 causing deadlock. Once patch0002 removed from pack, >>>>>> tests are running without any issue. >>>>>> >>>>>> >>>>>> On Thu, Nov 14, 2013 at 9:31 PM, Krishantha Samaraweera < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Integration tests hang at the early stage and following dead lock >>>>>>> situation detected in ESB Alpha pack. >>>>>>> >>>>>>> Name: http-nio-9443-exec-19 >>>>>>> State: BLOCKED on >>>>>>> org.apache.axis2.engine.AxisConfiguration@510b02daowned by: >>>>>>> pool-15-thread-1 >>>>>>> Total blocked: 2 Total waited: 29 >>>>>>> >>>>>>> Stack trace: >>>>>>> >>>>>>> >>>>>>> org.apache.axis2.engine.AxisConfiguration.removeService(AxisConfiguration.java:692) >>>>>>> >>>>>>> org.apache.synapse.config.SynapseConfiguration.removeProxyService(SynapseConfiguration.java:1037) >>>>>>> - locked org.apache.synapse.config.SynapseConfiguration@73e64d34 >>>>>>> >>>>>>> org.wso2.carbon.proxyadmin.service.ProxyServiceAdmin.deleteProxyService(ProxyServiceAdmin.java:398) >>>>>>> sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source) >>>>>>> >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>>> java.lang.reflect.Method.invoke(Method.java:597) >>>>>>> >>>>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) >>>>>>> >>>>>>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117) >>>>>>> >>>>>>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40) >>>>>>> >>>>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) >>>>>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>>>>> >>>>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) >>>>>>> >>>>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) >>>>>>> >>>>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java: >>>>>>> >>>>>>> >>>>>>> --------------------- >>>>>>> >>>>>>> Name: pool-15-thread-1 >>>>>>> State: BLOCKED on >>>>>>> org.apache.synapse.config.SynapseConfiguration@73e64d34 owned by: >>>>>>> http-nio-9443-exec-19 >>>>>>> Total blocked: 1 Total waited: 17 >>>>>>> >>>>>>> Stack trace: >>>>>>> >>>>>>> >>>>>>> org.apache.synapse.config.SynapseConfiguration.getDefinedSequences(SynapseConfiguration.java:289) >>>>>>> >>>>>>> org.apache.synapse.deployers.SequenceDeployer.updateSynapseArtifact(SequenceDeployer.java:120) >>>>>>> >>>>>>> org.wso2.carbon.sequences.SequenceDeploymentInterceptor.updateSynapseArtifact(SequenceDeploymentInterceptor.java:49) >>>>>>> >>>>>>> org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.resetDefaultSequence(AbstractSynapseArtifactDeployer.java:584) >>>>>>> >>>>>>> org.apache.synapse.deployers.SequenceDeployer.updatingFaultAndMainWithOriginal(SequenceDeployer.java:181) >>>>>>> >>>>>>> org.apache.synapse.deployers.SequenceDeployer.undeploy(SequenceDeployer.java:150) >>>>>>> >>>>>>> org.apache.axis2.deployment.DeploymentEngine.unDeploy(DeploymentEngine.java:909) >>>>>>> - locked org.wso2.carbon.core.CarbonAxisConfigurator@5497eb53 >>>>>>> >>>>>>> org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:143) >>>>>>> - locked org.wso2.carbon.core.CarbonAxisConfigurator@5497eb53 >>>>>>> >>>>>>> org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377) >>>>>>> >>>>>>> org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254) >>>>>>> >>>>>>> org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371) >>>>>>> >>>>>>> -- >>>>>>> Krishantha Samaraweera >>>>>>> Senior Technical Lead - Test Automation >>>>>>> Mobile: +94 77 7759918 >>>>>>> WSO2, Inc.; http://wso2.com/ >>>>>>> lean . enterprise . middlewear. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Miyuru Wanninayaka >>>>>> Technical Lead >>>>>> WSO2 Inc. : http://wso2.com >>>>>> >>>>>> Mobile : +94 77 209 9788 >>>>>> Blog : http://miyurudw.blogspot.com >>>>>> Flickr : http://www.flickr.com/photos/miyuru_daminda >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Miyuru Wanninayaka >>>>> Technical Lead >>>>> WSO2 Inc. : http://wso2.com >>>>> >>>>> Mobile : +94 77 209 9788 >>>>> Blog : http://miyurudw.blogspot.com >>>>> Flickr : http://www.flickr.com/photos/miyuru_daminda >>>>> >>>> >>>> >>>> >>>> -- >>>> Miyuru Wanninayaka >>>> Technical Lead >>>> WSO2 Inc. : http://wso2.com >>>> >>>> Mobile : +94 77 209 9788 >>>> Blog : http://miyurudw.blogspot.com >>>> Flickr : http://www.flickr.com/photos/miyuru_daminda >>>> >>> >>> >>> >>> -- >>> Miyuru Wanninayaka >>> Technical Lead >>> WSO2 Inc. : http://wso2.com >>> >>> Mobile : +94 77 209 9788 >>> Blog : http://miyurudw.blogspot.com >>> Flickr : http://www.flickr.com/photos/miyuru_daminda >>> >> >> >> >> -- >> Miyuru Wanninayaka >> Technical Lead >> WSO2 Inc. : http://wso2.com >> >> Mobile : +94 77 209 9788 >> Blog : http://miyurudw.blogspot.com >> Flickr : http://www.flickr.com/photos/miyuru_daminda >> > > > > -- > Krishantha Samaraweera > Senior Technical Lead - Test Automation > Mobile: +94 77 7759918 > WSO2, Inc.; http://wso2.com/ > lean . enterprise . middlewear. > -- Miyuru Wanninayaka Technical Lead WSO2 Inc. : http://wso2.com Mobile : +94 77 209 9788 Blog : http://miyurudw.blogspot.com Flickr : http://www.flickr.com/photos/miyuru_daminda
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
