Felix,
This situation looks like this to me:
1. First thread has the global lock and is trying to acquire a bundle
lock.
2. Second thread wants the global lock.
3. Third thread has two bundle locks (via registerService()) and is
blocked trying to get a third.
It is not clear to me who is holding the bundle lock that is blocking
the third thread. As far as I can guess, the first two threads do not
hold a bundle lock. Are you sure you have all of the threads that are
involved?
Regarding the comment about firing events in the code, it is relevant,
but I'd have to think about whether there was a way to refactor the
service registry to make it possible to fire the event externally.
-> richard
On 6/9/11 15:19, Felix Meschberger wrote:
Hi all,
We have a strange situation on a Framework 3.0.7 based system here which
is not reproducible on all platforms.
We can track down a system freeze/deadlock to three threads all
contending for bundle locks and the global lock:
"127.0.0.1 [1305717927375] GET /xxx.html HTTP/1.0" daemon prio=3
tid=0x0000000106203000 nid=0x2c01b in Object.wait() [0x00007ffe939f7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4712)
- locked<0x00007ffeabca9908> (a [Ljava.lang.Object;)
at
org.apache.felix.framework.Felix$FelixResolver.markBundleResolved(Felix.java:4220)
at
org.apache.felix.framework.Felix$FelixResolver.markResolvedModules(Felix.java:4200)
at
org.apache.felix.framework.Felix$FelixResolver.resolve(Felix.java:3999)
at org.apache.felix.framework.Felix.resolveBundle(Felix.java:3402)
at org.apache.felix.framework.Felix.loadBundleClass(Felix.java:1594)
at org.apache.felix.framework.BundleImpl.loadClass(BundleImpl.java:904)
at
org.apache.sling.commons.classloader.impl.PackageAdminClassLoader.findClass(PackageAdminClassLoader.java:150)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
- locked<0x00007ffecc57d040> (a
org.apache.sling.commons.classloader.impl.PackageAdminClassLoader)
at
org.apache.sling.commons.classloader.impl.PackageAdminClassLoader.loadClass(PackageAdminClassLoader.java:174)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
- locked<0x00007ffecc58b260> (a
org.apache.sling.jcr.classloader.internal.DynamicRepositoryClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.apache.jsp.apps.wci.components.page.page.page_jsp._jspService(page_jsp.java:170)
at
org.apache.sling.scripting.jsp.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.sling.scripting.jsp.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:419)
at
org.apache.sling.scripting.jsp.JspServletWrapperAdapter.service(JspServletWrapperAdapter.java:59)
at
org.apache.sling.scripting.jsp.JspScriptEngineFactory.callJsp(JspScriptEngineFactory.java:173)
at
org.apache.sling.scripting.jsp.JspScriptEngineFactory.access$100(JspScriptEngineFactory.java:84)
...
"ObservationManager" daemon prio=3 tid=0x000000010232e000 nid=0x30 in
Object.wait() [0x00007ffef24fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4785)
- locked<0x00007ffeabca9908> (a [Ljava.lang.Object;)
at org.apache.felix.framework.Felix.access$400(Felix.java:80)
at
org.apache.felix.framework.Felix$FelixResolver.resolve(Felix.java:4043)
at
org.apache.felix.framework.ModuleImpl.searchDynamicImports(ModuleImpl.java:1412)
at
org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:734)
at org.apache.felix.framework.ModuleImpl.access$400(ModuleImpl.java:71)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1768)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.apache.felix.framework.ModuleImpl.getClassByDelegation(ModuleImpl.java:645)
at
org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:544)
at
org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:275)
at
org.apache.felix.framework.Felix.getServiceReferences(Felix.java:2920)
at
org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:2970)
at
org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:309)
at
org.apache.felix.eventadmin.impl.handler.BlacklistingHandlerTasks.createHandlerTasks(BlacklistingHandlerTasks.java:102)
at
org.apache.felix.eventadmin.impl.EventAdminImpl.postEvent(EventAdminImpl.java:92)
at
org.apache.felix.eventadmin.impl.security.EventAdminSecurityDecorator.postEvent(EventAdminSecurityDecorator.java:77)
at
org.apache.sling.jcr.resource.internal.JcrResourceListener.sendOsgiEvent(JcrResourceListener.java:224)
at
org.apache.sling.jcr.resource.internal.JcrResourceListener.onEvent(JcrResourceListener.java:176)
at
org.apache.jackrabbit.core.observation.EventConsumer.consumeEvents(EventConsumer.java:248)
at
org.apache.jackrabbit.core.observation.ObservationDispatcher.run(ObservationDispatcher.java:161)
at java.lang.Thread.run(Thread.java:662)
"OsgiInstallerImpl" daemon prio=3 tid=0x0000000104f71800 nid=0x39 in
Object.wait() [0x00007ffef12fc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4712)
- locked<0x00007ffeabca9908> (a [Ljava.lang.Object;)
at org.apache.felix.framework.Felix.registerService(Felix.java:2813)
at
org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:251)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:449)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(AbstractComponentManager.java:460)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:1009)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:333)
at
org.apache.felix.scr.impl.manager.DependencyManager.serviceAdded(DependencyManager.java:283)
at
org.apache.felix.scr.impl.manager.DependencyManager.serviceChanged(DependencyManager.java:170)
at
org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:871)
at
org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:733)
at
org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3735)
at org.apache.felix.framework.Felix.access$000(Felix.java:80)
at org.apache.felix.framework.Felix$2.serviceChanged(Felix.java:722)
at
org.apache.felix.framework.ServiceRegistry.registerService(ServiceRegistry.java:107)
at org.apache.felix.framework.Felix.registerService(Felix.java:2847)
at
org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:251)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:449)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(AbstractComponentManager.java:460)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:1009)
at
org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:333)
at
org.apache.felix.scr.impl.manager.DependencyManager.serviceAdded(DependencyManager.java:283)
at
org.apache.felix.scr.impl.manager.DependencyManager.serviceChanged(DependencyManager.java:170)
at
org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:871)
at
org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:733)
at
org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3735)
at org.apache.felix.framework.Felix.access$000(Felix.java:80)
at org.apache.felix.framework.Felix$2.serviceChanged(Felix.java:722)
at
org.apache.felix.framework.ServiceRegistry.registerService(ServiceRegistry.java:107)
at org.apache.felix.framework.Felix.registerService(Felix.java:2847)
at
org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:251)
at
org.apache.sling.commons.classloader.impl.Activator.registerManagerFactory(Activator.java:78)
at
org.apache.sling.commons.classloader.impl.Activator.bundleChanged(Activator.java:122)
at
org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:807)
at
org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:729)
at
org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:610)
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:3724)
at org.apache.felix.framework.Felix.updateBundle(Felix.java:2030)
at org.apache.felix.framework.BundleImpl.update(BundleImpl.java:940)
at
org.apache.sling.installer.core.impl.tasks.BundleUpdateTask.execute(BundleUpdateTask.java:78)
at
org.apache.sling.installer.core.impl.OsgiInstallerImpl.executeTasks(OsgiInstallerImpl.java:522)
at
org.apache.sling.installer.core.impl.OsgiInstallerImpl.run(OsgiInstallerImpl.java:175)
Looking at the Framework source, particularly acquireBundleLock and
acquireGlobalLock I
cannot see where this "deadlock" can occur.
The only hint I have is a note in the Felix.registerService:
// TODO: CONCURRENCY - Reconsider firing event here, outside of the
// bundle lock.
I wonder whether this situation can be fixed with moving the service
registration
event ?
Regards
Felix
