Seems once proxy service deploy/undploy lock in axis2 config instead of
synapse config, no more deadlocks :)
I have to
- private AxisConfiguration axisConfiguration = null;
+ private AxisConfiguration axisConfiguration = new AxisConfiguration();
because in MultiXMLConfigurationBuilderTest, it creates a synapse config
without axisConfig which will cause NPE in synchronization block.
I'm currently running tests with these changes.
On Fri, Nov 15, 2013 at 10:37 AM, Miyuru Wanninayaka <[email protected]>wrote:
> May be for proxy service undeployment, synapse can lock on axis2 config
> instead of synapse config, which will avoid deadlocking.
>
>
> On Fri, Nov 15, 2013 at 10:22 AM, Miyuru Wanninayaka <[email protected]>wrote:
>
>> I have observed this few times even without patch0002.
>>
>> When analyzed deadloak flow it seems
>> hot deployment of main, fault sequences and deleting ESB artifacts from
>> admin services triggers in parallel.
>>
>> in hot deployment, axis2 thread first get a lock of axis2 configuration
>> and then calls synapse deployer.undeploy which will lock on synapse
>> configuration.
>> during admin service call, it will directly call synapse to undeploy
>> artifacts which will first lock on synapse config and in case of proxy, it
>> will cause related axis2 service to undeploy and try to acquire lock of
>> axis2 config which will become a deadlock.
>>
>> I have modified abstract ESBIntegrationTest.cleanup() method to wait few
>> seconds to let main and fault sequence to hot update. Seems deadlock does
>> not happens after that.
>>
>>
>> On Thu, Nov 14, 2013 at 10:17 PM, Miyuru Wanninayaka <[email protected]>wrote:
>>
>>> Seems patch0002 causing deadlock. Once patch0002 removed from pack,
>>> tests are running without any issue.
>>>
>>>
>>> On Thu, Nov 14, 2013 at 9:31 PM, Krishantha Samaraweera <
>>> [email protected]> wrote:
>>>
>>>> Integration tests hang at the early stage and following dead lock
>>>> situation detected in ESB Alpha pack.
>>>>
>>>> Name: http-nio-9443-exec-19
>>>> State: BLOCKED on org.apache.axis2.engine.AxisConfiguration@510b02daowned
>>>> by: pool-15-thread-1
>>>> Total blocked: 2 Total waited: 29
>>>>
>>>> Stack trace:
>>>>
>>>>
>>>> org.apache.axis2.engine.AxisConfiguration.removeService(AxisConfiguration.java:692)
>>>>
>>>> org.apache.synapse.config.SynapseConfiguration.removeProxyService(SynapseConfiguration.java:1037)
>>>> - locked org.apache.synapse.config.SynapseConfiguration@73e64d34
>>>>
>>>> org.wso2.carbon.proxyadmin.service.ProxyServiceAdmin.deleteProxyService(ProxyServiceAdmin.java:398)
>>>> sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> java.lang.reflect.Method.invoke(Method.java:597)
>>>>
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>>
>>>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
>>>>
>>>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>>>>
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>>
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:
>>>>
>>>>
>>>> ---------------------
>>>>
>>>> Name: pool-15-thread-1
>>>> State: BLOCKED on
>>>> org.apache.synapse.config.SynapseConfiguration@73e64d34 owned by:
>>>> http-nio-9443-exec-19
>>>> Total blocked: 1 Total waited: 17
>>>>
>>>> Stack trace:
>>>>
>>>>
>>>> org.apache.synapse.config.SynapseConfiguration.getDefinedSequences(SynapseConfiguration.java:289)
>>>>
>>>> org.apache.synapse.deployers.SequenceDeployer.updateSynapseArtifact(SequenceDeployer.java:120)
>>>>
>>>> org.wso2.carbon.sequences.SequenceDeploymentInterceptor.updateSynapseArtifact(SequenceDeploymentInterceptor.java:49)
>>>>
>>>> org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.resetDefaultSequence(AbstractSynapseArtifactDeployer.java:584)
>>>>
>>>> org.apache.synapse.deployers.SequenceDeployer.updatingFaultAndMainWithOriginal(SequenceDeployer.java:181)
>>>>
>>>> org.apache.synapse.deployers.SequenceDeployer.undeploy(SequenceDeployer.java:150)
>>>>
>>>> org.apache.axis2.deployment.DeploymentEngine.unDeploy(DeploymentEngine.java:909)
>>>> - locked org.wso2.carbon.core.CarbonAxisConfigurator@5497eb53
>>>>
>>>> org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:143)
>>>> - locked org.wso2.carbon.core.CarbonAxisConfigurator@5497eb53
>>>>
>>>> org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
>>>>
>>>> org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
>>>>
>>>> org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
>>>>
>>>> --
>>>> Krishantha Samaraweera
>>>> Senior Technical Lead - Test Automation
>>>> Mobile: +94 77 7759918
>>>> WSO2, Inc.; http://wso2.com/
>>>> lean . enterprise . middlewear.
>>>>
>>>
>>>
>>>
>>> --
>>> Miyuru Wanninayaka
>>> Technical Lead
>>> WSO2 Inc. : http://wso2.com
>>>
>>> Mobile : +94 77 209 9788
>>> Blog : http://miyurudw.blogspot.com
>>> Flickr : http://www.flickr.com/photos/miyuru_daminda
>>>
>>
>>
>>
>> --
>> Miyuru Wanninayaka
>> Technical Lead
>> WSO2 Inc. : http://wso2.com
>>
>> Mobile : +94 77 209 9788
>> Blog : http://miyurudw.blogspot.com
>> Flickr : http://www.flickr.com/photos/miyuru_daminda
>>
>
>
>
> --
> Miyuru Wanninayaka
> Technical Lead
> WSO2 Inc. : http://wso2.com
>
> Mobile : +94 77 209 9788
> Blog : http://miyurudw.blogspot.com
> Flickr : http://www.flickr.com/photos/miyuru_daminda
>
--
Miyuru Wanninayaka
Technical Lead
WSO2 Inc. : http://wso2.com
Mobile : +94 77 209 9788
Blog : http://miyurudw.blogspot.com
Flickr : http://www.flickr.com/photos/miyuru_daminda
Index: src/main/java/org/apache/synapse/config/SynapseConfiguration.java
===================================================================
--- src/main/java/org/apache/synapse/config/SynapseConfiguration.java
(revision 191088)
+++ src/main/java/org/apache/synapse/config/SynapseConfiguration.java
(working copy)
@@ -136,7 +136,7 @@
/**
* Hold reference to the Axis2 ConfigurationContext
*/
- private AxisConfiguration axisConfiguration = null;
+ private AxisConfiguration axisConfiguration = new AxisConfiguration();
/**
* Save the path to the configuration file loaded, to save it later if
@@ -996,13 +996,16 @@
* @param proxy the Proxy service instance
*/
public synchronized void addProxyService(String name, ProxyService proxy) {
- if (!proxyServices.containsKey(name)) {
- proxyServices.put(name, proxy);
- for (SynapseObserver o : observers) {
- o.proxyServiceAdded(proxy);
+ synchronized (this.axisConfiguration) {
+ log.info(axisConfiguration == null);
+ if (!proxyServices.containsKey(name)) {
+ proxyServices.put(name, proxy);
+ for (SynapseObserver o : observers) {
+ o.proxyServiceAdded(proxy);
+ }
+ } else {
+ handleException("Duplicate proxy service by the name : " +
name);
}
- } else {
- handleException("Duplicate proxy service by the name : " + name);
}
}
@@ -1022,26 +1025,28 @@
*
* @param name of the Proxy Service to be deleted
*/
- public synchronized void removeProxyService(String name) {
- ProxyService proxy = proxyServices.get(name);
- if (proxy == null) {
- handleException("Unknown proxy service for name : " + name);
- } else {
- try {
- if (getAxisConfiguration().getServiceForActivation(name) !=
null) {
- if (getAxisConfiguration().getServiceForActivation(name)
- .isActive()) {
- getAxisConfiguration().getService(name)
- .setActive(false);
+ public void removeProxyService(String name) {
+ synchronized (this.axisConfiguration) {
+ ProxyService proxy = proxyServices.get(name);
+ if (proxy == null) {
+ handleException("Unknown proxy service for name : " + name);
+ } else {
+ try {
+ if (getAxisConfiguration().getServiceForActivation(name)
!= null) {
+ if
(getAxisConfiguration().getServiceForActivation(name)
+ .isActive()) {
+ getAxisConfiguration().getService(name)
+ .setActive(false);
+ }
+ getAxisConfiguration().removeService(name);
}
- getAxisConfiguration().removeService(name);
+ proxyServices.remove(name);
+ for (SynapseObserver o : observers) {
+ o.proxyServiceRemoved(proxy);
+ }
+ } catch (AxisFault axisFault) {
+ handleException(axisFault.getMessage());
}
- proxyServices.remove(name);
- for (SynapseObserver o : observers) {
- o.proxyServiceRemoved(proxy);
- }
- } catch (AxisFault axisFault) {
- handleException(axisFault.getMessage());
}
}
}
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
