Hi Ivan, Here's my current thinking on this. I could easily be wrong about many aspects of this :-) and I'm not done...
I think there are two kinds of problems to avoid here. The easy one is to synchronize access to our configuration tracking objects. I think we can do this by synchronizing on configurations for all access to configurations, configurationModel, and reloadingConfiguration. This is really similar to your commit, but without the new object and (I think) synchronizing access to configurationModel. The harder part is that we don't want two threads trying to change the state of a configuration at the same time. So I think we need an Artifact to lock object map and when we are changing the state of a configuration we secure the appropriate lock object and synchronize on it while the state change is happening. Possibly the ConfigurationStatus in the configurationModel can be the lock object, I'm not sure if the lifecycle is correct for the status object to be the lock object. I think we also need to check before each state change (after getting the lock) what state the configuration is actually in. Finally I don't think a single reloadingConfiguration will work any more. I'm not sure if the map of lock objects will work for this too or if we need something else. Many thanks for fixing at least the initial problem so quickly! david jencks On Jan 29, 2011, at 5:59 PM, Ivan wrote: > For this single issue, another lock on the loadingConfiguration is required, > too. I attached a proposed patch on GERONIMO-5782 > But that is only for this one, if no objection, I would commit it soon. As > many of jsf/ejb cases are blocked by this. Personally, I always get nervous > while introducing more locks in one classes :-) And I guess that we might > check some other bundle extender codes. I am still working on another dead > lock on the WAB deployment, it uses a new thread for the deployment, too. > Thanks. > > 2011/1/29 David Jencks <[email protected]> > We might have to make the SimpleConfigurationManager less synchronized or > better synchronized. I think the first rule of avoiding deadlocks is to not > call unknown code from within a synchronized block. We are definitely > calling such code here, we're starting the entire configuration from inside a > synchronized block. > > Basically I think we need to review the SimpleConfigurationManager code and > make the locking finer grained and avoid locking when calling out. For > instance we could synchronize getConfiguration on the artifact-configuration > map rather than the configuration manager object. > > I'll try to look at this in the next couple days. Any other analysis or > suggestions would be more than welcome. > > thanks > david jencks > > On Jan 28, 2011, at 7:45 PM, Ivan wrote: > >> The related stack should be, it is caused by the new thread used by Tomcat >> for listener initialization, MyFaces listener called some functions in the >> SimpleConfigurationManager >> I do not have a good idea so far, maybe we should try to disable the >> function in Tomcat ? Or any invocation on the configuraiton manager in any >> listener/servlet init will cause the dead lock. >> >> ---> >> "Thread-58" daemon prio=10 tid=0x0000000040ac5000 nid=0x3ff7 waiting for >> monitor entry [0x00007f6cdd948000] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at >> org.apache.geronimo.kernel.config.SimpleConfigurationManager.getConfiguration(SimpleConfigurationManager.java:264) >> - waiting to lock <0x00007f6cf84b1570> (a >> org.apache.geronimo.kernel.config.EditableKernelConfigurationManager) >> at >> org.apache.geronimo.naming.reference.ConfigurationAwareReference.getConfiguration(ConfigurationAwareReference.java:61) >> at >> org.apache.geronimo.naming.reference.ConfigurationAwareReference.resolveTargetName(ConfigurationAwareReference.java:79) >> at >> org.apache.geronimo.naming.reference.GBeanReference.getContent(GBeanReference.java:46) >> at >> org.apache.xbean.naming.context.ContextUtil.resolve(ContextUtil.java:61) >> at >> org.apache.xbean.naming.context.ContextUtil$ReadOnlyBinding.getObject(ContextUtil.java:199) >> at >> org.apache.xbean.naming.context.ContextFederation.getFederatedBindings(ContextFederation.java:119) >> at >> org.apache.xbean.naming.context.AbstractFederatedContext.getBindings(AbstractFederatedContext.java:99) >> at >> org.apache.xbean.naming.context.AbstractFederatedContext.getBinding(AbstractFederatedContext.java:86) >> at >> org.apache.xbean.naming.context.AbstractContext.lookup(AbstractContext.java:133) >> at >> org.apache.xbean.naming.context.AbstractContext.lookup(AbstractContext.java:605) >> at >> org.apache.geronimo.naming.java.RootContext.lookup(RootContext.java:55) >> at >> org.apache.aries.jndi.DelegateContext.lookup(DelegateContext.java:148) >> at javax.naming.InitialContext.lookup(InitialContext.java:392) >> at >> org.apache.myfaces.application.ApplicationImpl.getProjectStage(ApplicationImpl.java:713) >> at >> org.apache.webbeans.jsf.OwbApplication.getProjectStage(OwbApplication.java:443) >> at >> javax.faces.context.FacesContext.isProjectStage(FacesContext.java:411) >> at >> org.apache.myfaces.config.FacesConfigurator.configureLifecycle(FacesConfigurator.java:1011) >> at >> org.apache.myfaces.config.FacesConfigurator.configure(FacesConfigurator.java:465) >> at >> org.apache.myfaces.webapp.AbstractFacesInitializer.buildConfiguration(AbstractFacesInitializer.java:303) >> at >> org.apache.myfaces.webapp.Jsp21FacesInitializer.initContainerIntegration(Jsp21FacesInitializer.java:73) >> at >> org.apache.myfaces.webapp.AbstractFacesInitializer.initFaces(AbstractFacesInitializer.java:126) >> at >> org.apache.myfaces.webapp.StartupServletContextListener.contextInitialized(StartupServletContextListener.java:110) >> at >> org.apache.geronimo.myfaces.webapp.GeronimoStartupServletContextListener.contextInitialized(GeronimoStartupServletContextListener.java:55) >> at >> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4521) >> at >> org.apache.catalina.core.StandardContext$1.call(StandardContext.java:5004) >> at >> org.apache.catalina.core.StandardContext$1.call(StandardContext.java:4999) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> "RMI TCP Connection(6)-172.16.21.3" daemon prio=10 tid=0x0000000040c8e800 >> nid=0x3ff2 waiting on condition [0x00007f6cdf189000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00007f6cfbc53170> (a >> java.util.concurrent.FutureTask$Sync) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) >> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218) >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> at >> org.apache.tomcat.util.threads.DedicatedThreadExecutor.execute(DedicatedThreadExecutor.java:56) >> at >> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4998) >> - locked <0x00007f6cfbc53250> (a >> org.apache.geronimo.tomcat.GeronimoStandardContext) >> at >> org.apache.geronimo.tomcat.GeronimoStandardContext.access$101(GeronimoStandardContext.java:105) >> at >> org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java:728) >> at >> org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke(GeronimoBeforeAfterValve.java:47) >> at >> org.apache.geronimo.tomcat.valve.ProtectedTargetValve.invoke(ProtectedTargetValve.java:53) >> at >> org.apache.geronimo.tomcat.GeronimoStandardContext.startInternal(GeronimoStandardContext.java:463) >> at >> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >> - locked <0x00007f6cfbc53250> (a >> org.apache.geronimo.tomcat.GeronimoStandardContext) >> at >> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:812) >> at >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:787) >> at >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) >> at >> org.apache.geronimo.tomcat.TomcatContainer.addContext(TomcatContainer.java:310) >> at >> org.apache.geronimo.tomcat.TomcatWebAppContext.doStart(TomcatWebAppContext.java:568) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:975) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:271) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:105) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:546) >> at >> org.apache.geronimo.gbean.runtime.GBeanDependency.attemptFullStart(GBeanDependency.java:110) >> - locked <0x00007f6cfbae5210> (a >> org.apache.geronimo.gbean.runtime.GBeanDependency) >> at >> org.apache.geronimo.gbean.runtime.GBeanDependency.addTarget(GBeanDependency.java:145) >> - locked <0x00007f6cfbae5210> (a >> org.apache.geronimo.gbean.runtime.GBeanDependency) >> at >> org.apache.geronimo.gbean.runtime.GBeanDependency$1.running(GBeanDependency.java:119) >> at >> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.fireRunningEvent(BasicLifecycleMonitor.java:175) >> at >> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.access$300(BasicLifecycleMonitor.java:44) >> at >> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor$RawLifecycleBroadcaster.fireRunningEvent(BasicLifecycleMonitor.java:253) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:301) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:105) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:546) >> at >> org.apache.geronimo.gbean.runtime.GBeanDependency.attemptFullStart(GBeanDependency.java:110) >> - locked <0x00007f6cfbadedf0> (a >> org.apache.geronimo.gbean.runtime.GBeanDependency) >> at >> org.apache.geronimo.gbean.runtime.GBeanDependency.addTarget(GBeanDependency.java:145) >> - locked <0x00007f6cfbadedf0> (a >> org.apache.geronimo.gbean.runtime.GBeanDependency) >> at >> org.apache.geronimo.gbean.runtime.GBeanDependency$1.running(GBeanDependency.java:119) >> at >> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.fireRunningEvent(BasicLifecycleMonitor.java:175) >> at >> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.access$300(BasicLifecycleMonitor.java:44) >> at >> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor$RawLifecycleBroadcaster.fireRunningEvent(BasicLifecycleMonitor.java:253) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:301) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:105) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java:127) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:560) >> at >> org.apache.geronimo.kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:386) >> at >> org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:460) >> at >> org.apache.geronimo.kernel.config.KernelConfigurationManager.start(KernelConfigurationManager.java:226) >> at >> org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:698) >> - locked <0x00007f6cf84b1570> (a >> org.apache.geronimo.kernel.config.EditableKernelConfigurationManager) >> at >> org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:677) >> - locked <0x00007f6cf84b1570> (a >> org.apache.geronimo.kernel.config.EditableKernelConfigurationManager) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34) >> at >> org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:131) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:872) >> at >> org.apache.geronimo.kernel.basic.BasicKernel.invoke(BasicKernel.java:245) >> at org.apache.geronimo.kernel.KernelGBean.invoke(KernelGBean.java:344) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34) >> at >> org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:131) >> at >> org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:872) >> at >> org.apache.geronimo.kernel.basic.BasicKernel.invoke(BasicKernel.java:245) >> at >> org.apache.geronimo.system.jmx.MBeanGBeanBridge.invoke(MBeanGBeanBridge.java:172) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) >> at >> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) >> at >> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) >> at >> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) >> at >> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367) >> at >> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) >> at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >> at sun.rmi.transport.Transport$1.run(Transport.java:159) >> at java.security.AccessController.doPrivileged(Native Method) >> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >> at >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> <-- >> >> 2011/1/29 Shawn Jiang <[email protected]> >> See my comments to the tck results in tck list and the log in GERONIMO-5782. >> >> -- >> Shawn >> >> >> >> -- >> Ivan > > > > > -- > Ivan
