Hi Karl et al, I've written a script which detects cyclic dependencies from the deployed jars manifests. I'm finding rather a lot in Felix and Knopflerfish jars. For example, seemingly the Gogo bundles all seem to like to import packages that they provide - is this likely to be a problem, or is the fact that they're from the same bundle get recognised in Felix? Best, Dan.
On 25 Nov 2013, at 22:01, Karl Pauls wrote: > Are you sure you don't have any cyclic dependencies between your bundles > anymore (maybe you could send me your manifests too)? > > regards, > > Karl > > > On Mon, Nov 25, 2013 at 10:10 PM, Daniel McGreal <[email protected]> wrote: > >> Hi Felix people, >> >> I've come back to revisit this issue, where threads are deadblocking >> during classloading when running under JamVM and Felix 4.3.1. Having now >> completed other deployments, it's now critical for me to get this working >> ASAP, therefore any help you can offer is tremendously gratefully received. >> >> I've removed a cyclic dependency, as Karl suggested, which I believe >> reduced the occurrence rate. I've also started to test without >> Retrotranslation, which hasn't had an effect. >> >> A common element to the deadlock is SCR: >> >> "FelixStartLevel" (daemon) 0x1aaf9a0 priority: 5 tid: 0x7f69ce3a8700 id: 6 >> state: BLOCKED (5) >> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:580) >> at >> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1948) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:293) >> at >> org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(BundleWiringImpl.java:1374) >> at >> org.apache.felix.framework.BundleWiringImpl.searchImports(BundleWiringImpl.java:1553) >> at >> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1484) >> at >> org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75) >> at >> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:293) >> at java.lang.VMClass.getDeclaredMethods(Native method) >> at java.lang.Class.getDeclaredMethods(Class.java:536) >> at java.lang.Class.getDeclaredMethod(Class.java:506) >> at >> org.apache.felix.scr.impl.helper.BaseMethod.getMethod(BaseMethod.java:328) >> at >> org.apache.felix.scr.impl.helper.BindMethod.getMethod(BindMethod.java:38) >> at >> org.apache.felix.scr.impl.helper.BindMethod.getServiceReferenceMethod(BindMethod.java:350) >> at >> org.apache.felix.scr.impl.helper.BindMethod.doFindMethod(BindMethod.java:99) >> at >> org.apache.felix.scr.impl.helper.BaseMethod.findMethod(BaseMethod.java:185) >> at >> org.apache.felix.scr.impl.helper.BaseMethod.access$400(BaseMethod.java:37) >> at >> org.apache.felix.scr.impl.helper.BaseMethod$NotResolved.resolve(BaseMethod.java:555) >> at >> org.apache.felix.scr.impl.helper.BaseMethod$NotResolved.methodExists(BaseMethod.java:578) >> at >> org.apache.felix.scr.impl.helper.BaseMethod.methodExists(BaseMethod.java:510) >> at >> org.apache.felix.scr.impl.helper.BindMethod.methodExists(BindMethod.java:38) >> at >> org.apache.felix.scr.impl.helper.BindMethod.getServiceObject(BindMethod.java:560) >> at >> org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1091) >> at >> org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:768) >> at >> org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:1492) >> at >> org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:550) >> at >> org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:261) >> at >> org.apache.felix.scr.impl.config.ImmediateComponentHolder.enableComponents(ImmediateComponentHolder.java:328) >> at >> org.apache.felix.scr.impl.BundleComponentActivator.initialize(BundleComponentActivator.java:158) >> at >> org.apache.felix.scr.impl.BundleComponentActivator.<init>(BundleComponentActivator.java:113) >> at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:261) >> at >> org.apache.felix.scr.impl.Activator.loadAllComponents(Activator.java:199) >> at org.apache.felix.scr.impl.Activator.start(Activator.java:108) >> at >> org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:645) >> at org.apache.felix.framework.Felix.activateBundle(Felix.java:2146) >> at org.apache.felix.framework.Felix.startBundle(Felix.java:2064) >> at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1291) >> at >> org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304) >> at java.lang.Thread.run(Thread.java:745) >> >> After taking this trace, I changed SCR to start before the bundles it >> should load, the deadlock still happens with a similar stack but now from >> org.apache.felix.scr.impl.Activator.bundleChanged. In fact, having tried to >> avoid the issue by using start levels and loading bundles in a natural >> order (previously everything was getting started via auto-deploy), the >> deadlock occurs every time (whereas it was down to more like 1 in 20 with >> auto-deploy). >> >> A couple of examples of the other blocked threads are attached. >> >> >> >> >> In every case, FelixStartLevel blocks at ClassLoader.java:580 (via SCR >> and BundleWiringImpl.java:1948 loadClass) and BundleWiringImpl.java:2038 >> findClass (though that line seems inconspicuous - if (!hooks.isEmpty()), >> I don't readily see how a block could happen there - hooks is a >> TreeMap?). I added some debug output to BundleWiringImpl.java findClass and >> loadClass methods (no debugger with JamVM): >> >> >> @Override >> protected Class loadClass(final String name, final booleanresolve) >> throws ClassNotFoundException >> { >> System.out.println(getBundle().getSymbolicName() +" loadClass "+ >> name +", "+ resolve); >> >> ------ >> >> >> @Override >> protected Class findClass(final String name) >> throwsClassNotFoundException >> { >> System.out.println(getBundle().getSymbolicName() +" findClass "+ >> name); >> >> >> >> The output I get, leads me to believe that the deadlock is not particular >> to a bundle or class. For example (the last findClass and loadClass >> highlighted): >> >> ... >> org.apache.felix.gogo.command *findClass* org.osgi.service.log.LogEntry >> obs.logging.amqp loadClass >> com.redbite.rededge.obs.interfaces.deployment.DeviceRegistry, false >> obs.interfaces.deployment loadClass >> com.redbite.rededge.obs.interfaces.deployment.DeviceRegistry, false >> obs.logging.amqp loadClass com.rabbitmq.client.Channel, false >> com.redbite.osgi.com.rabbitmq.amqp-client >> *loadClass*com.rabbitmq.client.Channel, false >> >> >> Do you think I am interpreting this correctly? Now though, as well as >> blocking on [what used to be] line 2038 (if (!hooks.isEmpty())), it >> actually now blocks on the line I've added, >> System.out.println(getBundle().getSymbolicName() >> +" findClass "+ name); >> >> Any ideas on what I can do to pursue this further, gratefully received. >> Best, Dan. >> >> On 12 Nov 2013, at 11:18, Karl Pauls wrote: >> >> On the retrotranslator bundles, they are made available by specifying a >> >> dynamic-import which is added to the manifest during retrotranslation (so >> >> that our code/libraries can run on). >> >> >> >> I would try to change that and either have that bundles each contain their >> own copy or to not have them mention it at all in the manifest and just put >> them on the class path of the framework and boot delegate. >> >> >> Unfortunately, running unretrotranslated does not help. >> >> >> >> Hm, in that case something else must be going on - however, in your stack >> trace it clearly all happens inside the retrotranslate part so you might >> have to give me a stack trace of the unretrotranslated one as well. >> >> regards, >> >> Karl >> >> p.s.: feel free to also share your manifests if that is possible >> >> >> Thanks a lot for your help, >> >> Dan. >> >> >> On 12 Nov 2013, at 09:51, Karl Pauls wrote: >> >> >> Well, my guess would be that you are running into some form of a circular >> >> class load issue. The point is that the class loader in java is >> >> synchronized which my lead to deadlocks if you have a cycle in your class >> >> loaders. This is a long standing issue with frameworks like OSGi which >> >> allow for cycles in the dependencies. >> >> >> I don't know enough about your app to say for sure nor to help you with >> >> this but look at the dependencies of your bundles and see whether they >> >> create a cycle (i.e., Bundle A depends via some import on Bundle B which >> >> depends via some import on Bundle C which depends via some import on >> >> Bundle >> >> A or something similar). In your case, it might be even more complicated >> >> as >> >> the retrotranslator seems to do some class loading magic by itself which >> >> might be responsible for the issue - how do you make the >> >> net.sf.retrotranslator.* >> >> packages available to the bundles? >> >> >> regards, >> >> >> Karl >> >> >> >> On Fri, Nov 8, 2013 at 10:44 PM, Daniel McGreal <[email protected]> >> >> wrote: >> >> >> Hello Felix developer community, >> >> >> I am having an interesting (and critical) problem running my application >> >> with Felix in an embedded environment where an alternate JVM, JamVM is >> >> necessary. >> >> >> The deadlock is non-deterministic with the deadlocked stack traces >> >> blocking at odd places (though they might not seem odd to people used to >> >> messing with class loading - but a block on new XXX() to me is quite the >> >> novelty). An example of the relevant parts of the thread stack is in >> >> FelixJamVMDeadlockStackTrace.txt, attached. It shows a lock between: >> >> >> >> - >> >> - java.lang.ClassLoader.findLoadedClass(ClassLoader.java:580) looks >> >> like: >> >> >> protected final synchronized Class<?> findLoadedClass(String name) >> >> { >> >> checkInitialized(); >> >> return VMClassLoader.findLoadedClass(this, name); >> >> } >> >> >> where line 580 is checkInitialized() which throws an exception if >> >> isInitialized is false. >> >> >> - >> >> >> net.sf.retrotranslator.runtime.asm.ClassReader.readConst(ClassReader.java:1591) >> >> looks like: >> >> >> public Object readConst(final int item, final char[] buf) { >> >> int index = items[item]; >> >> switch (b[index - 1]) { >> >> case ClassWriter.INT: >> >> return new Integer(readInt(index)); >> >> case ClassWriter.FLOAT: >> >> return new >> >> Float(Float.intBitsToFloat(readInt(index))); >> >> case ClassWriter.LONG: >> >> return new Long(readLong(index)); >> >> case ClassWriter.DOUBLE: >> >> return >> >> newDouble(Double.longBitsToDouble(readLong(index))); >> >> case ClassWriter.CLASS: >> >> String s = readUTF8(index, buf); >> >> return Type.getType(s.charAt(0) == '[' ? s : "L" + s + >> >> ";"); >> >> // case ClassWriter.STR: >> >> default: >> >> return readUTF8(index, buf); >> >> } >> >> } >> >> where line 1591 is return new Integer(readInt(index)); >> >> >> - >> >> - >> >> - >> >> >> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.findClass(BundleWiringImpl.java:2038) >> >> is a big method, where line 2038 looks inconspicuously like: >> >> >> if (!hooks.isEmpty()) >> >> >> >> Gogo and Retrotranslator are red herrings, without them in the >> >> application >> >> the deadlock just happens somewhere else. >> >> >> I don't even vaguely understand how, but neither Eclipse nor >> >> Knopflerfish >> >> suffer from this problem. I would, however, much rather use Felix. >> >> >> Please let me know if there's any further information I can provide. >> >> >> Best, Dan. >> >> >> >> >> >> >> >> >> >> -- >> >> Karl Pauls >> >> [email protected] >> >> http://twitter.com/karlpauls >> >> http://www.linkedin.com/in/karlpauls >> >> https://profiles.google.com/karlpauls >> >> >> >> >> >> -- >> Karl Pauls >> [email protected] >> http://twitter.com/karlpauls >> http://www.linkedin.com/in/karlpauls >> https://profiles.google.com/karlpauls >> >> >> >> > > > -- > Karl Pauls > [email protected] > http://twitter.com/karlpauls > http://www.linkedin.com/in/karlpauls > https://profiles.google.com/karlpauls
