Well, it is good to hear that both of our sanity is intact; I'm not sure how I even thought to try changing around the way logging appeared on my classpath in the first place, but I'm glad I did.
We have the same basic setup: trident topologies, custom logback XML on the classpath and an appender-ref (probably not so strange a setup, after all). For me, removing the appender-ref element does not resolve the issue, although removing the XML file entirely does. What is weird here is that I get the same issue if I have no slf4j backend on my classpath...or two. So it seems like in order for things to go smoothly, slf4j needs to initialize properly and in a very specific way. I know slf4j does some tricky classpath stuff to do its thing, and I have no idea about Clojure implementation details...but it's a clue. I'm glad to hear you've not seen this in 0.9.2-SNAPSHOT since STORM-290. I'd try it out right now if I weren't in the process of "mvn release:prepare"ing all my internal projects...although I look forward to 0.9.2 to go final for a number of reasons now. Thanks for the keen eye, Taylor. On Tue, Apr 22, 2014 at 12:21 PM, P. Taylor Goetz <[email protected]> wrote: > > I have seen this as well and thought I was going nuts. In my testing I > could reliably reproduce it in local mode against 0.9.1-incubating. > > What I noticed in my testing: > - It only occurred when a custom logback.xml was on the class path. > - More specifically, it only happened when there was an “appender-ref” > child element in the “root”. Commenting out the “appender-ref” element made > it go away. > - It only happened with Trident topologies. It did not happen with regular > storm topologies. > > It is clearly a classloader issue, but I have no idea what the root cause > is. My theory is that it is related to the fact that in 0.9.1 we had both > the clojure source code as well as the AOT-compiled clojure classes in the > storm-core jar. In 0.9.2-SNAPSHOT (latest master branch), we no longer > package the clojure source code alongside the AOT-compiled classes. In > 0.9.2, with the patch for STORM-290 applied, this problem goes away. Not > absolutely sure why, but I’m glad it does. ;) > > - Taylor > > On Apr 21, 2014, at 9:09 PM, Adam Lewis <[email protected]> wrote: > > The tests are in the test tree (which I don't think eclipse cares about, > but should get treated properly on the command line). The scope wasn't > provided, it was defaulted to runtime...and I've now corrected that and it > doesn't help. > > Strangely, if I explicitly exclude logback-classic within the storm > dependency, it does work as long as I also add a dependency on > slf4j-simple...so I am still seeing weird logger induced classpath > issues...so this is what is working for me: > > <dependency> > <groupId>org.apache.storm</groupId> > <artifactId>storm-core</artifactId> > <exclusions> > <exclusion> > <artifactId>logback-classic</artifactId> > <groupId>ch.qos.logback</groupId> > </exclusion> > </exclusions> > </dependency> > <dependency> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-simple</artifactId> > </dependency> > > WEIRD! > > > > On Mon, Apr 21, 2014 at 8:54 PM, Jon Logan <[email protected]> wrote: > >> Are your maven scopes right? The scope of the Storm dependency should be >> provided -- not runtime. Also be sure that your main method / unit test is >> under your test/ classpath, not your main/ classpath. >> >> >> On Mon, Apr 21, 2014 at 8:49 PM, Adam Lewis <[email protected]> wrote: >> >>> Are there other things that could cause this error? Since upgrading to >>> 0.9.1-incubating, I've hit it twice. The first time I resolved it (in one >>> project) by fixing an issue where two slf4j bindings were on the classpath >>> together (strange, but it worked)...now I'm hitting the problem again in a >>> different project and can't figure out what is causing the problem. This >>> is for a test which is submitting a topology to a LocalCluster; the full >>> trace follows (happens launching JUnit from Eclipse and from Maven command >>> line) >>> >>> java.lang.RuntimeException: java.lang.ClassNotFoundException: >>> backtype.storm.daemon.nimbus$normalize_conf$get_merged_conf_val__3916$fn__3917 >>> at backtype.storm.utils.Utils.deserialize(Utils.java:88) >>> at backtype.storm.daemon.nimbus$read_storm_conf.invoke(nimbus.clj:89) >>> at backtype.storm.daemon.nimbus$start_storm.invoke(nimbus.clj:724) >>> at >>> backtype.storm.daemon.nimbus$eval3974$exec_fn__1459__auto__$reify__3987.submitTopologyWithOpts(nimbus.clj:962) >>> at >>> backtype.storm.daemon.nimbus$eval3974$exec_fn__1459__auto__$reify__3987.submitTopology(nimbus.clj:971) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) >>> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) >>> at backtype.storm.testing$submit_local_topology.invoke(testing.clj:253) >>> at >>> backtype.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:34) >>> at backtype.storm.LocalCluster.submitTopology(Unknown Source) >>> at >>> com.acuitysds.trident.TestTimeseriesAssembly.testBasicTopology(TestTimeseriesAssembly.java:108) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) >>> at >>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) >>> at >>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) >>> at >>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) >>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) >>> at >>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) >>> at >>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) >>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) >>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) >>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) >>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) >>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) >>> at org.junit.runners.ParentRunner.run(ParentRunner.java:309) >>> at >>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) >>> at >>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) >>> at >>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) >>> at >>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) >>> at >>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) >>> at >>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) >>> Caused by: java.lang.ClassNotFoundException: >>> backtype.storm.daemon.nimbus$normalize_conf$get_merged_conf_val__3916$fn__3917 >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> at java.lang.Class.forName0(Native Method) >>> at java.lang.Class.forName(Class.java:270) >>> at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:623) >>> at >>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1610) >>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1704) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >>> at backtype.storm.utils.Utils.deserialize(Utils.java:82) >>> ... 37 more >>> >>> >>> >>> On Mon, Mar 10, 2014 at 12:34 PM, James Hardy <[email protected]>wrote: >>> >>>> It was indeed a classpath/maven issue. Thank you Nathan! >>>> >>>> >>>> On Mon, Mar 10, 2014 at 2:36 PM, Nathan Leung <[email protected]>wrote: >>>> >>>>> These appear to classes generated from clojure. Are you building >>>>> against the same version of storm that you are using to run the >>>>> application? >>>>> >>>>> >>>>> On Mon, Mar 10, 2014 at 8:07 AM, James Hardy <[email protected]>wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> I'm hoping someone can help. I've been learning Storm for the past >>>>>> few hours and everything has been great however I'm getting >>>>>> a java.lang.ClassNotFoundException: >>>>>> backtype.storm.daemon.nimbus$normalize_conf$get_merged_conf_val__3916$fn__3917 >>>>>> exception when trying to submit my topology to a local cluster. >>>>>> >>>>>> I have a simple spout which emits the string "go" and >>>>>> the ExclamationBolt taken from the examples. I wanted to create a >>>>>> pipeline >>>>>> of bolts and so created a topology: >>>>>> >>>>>> TopologyBuilder builder = new TopologyBuilder(); >>>>>> builder.setSpout("test", new Spout()); >>>>>> builder.setBolt("ex1", new ExclamationBolt()).shuffleGrouping("test"); >>>>>> builder.setBolt("ex2", new ExclamationBolt()).shuffleGrouping("ex1"); >>>>>> >>>>>> This works fine no problem at all however when I add another bolt I >>>>>> get the "Topology submission exception": >>>>>> >>>>>> TopologyBuilder builder = new TopologyBuilder(); >>>>>> builder.setSpout("test", new Spout()); >>>>>> builder.setBolt("ex1", new ExclamationBolt()).shuffleGrouping("test"); >>>>>> builder.setBolt("ex2", new ExclamationBolt()).shuffleGrouping("ex1"); >>>>>> builder.setBolt("ex3", new ExclamationBolt()).shuffleGrouping("ex2"); >>>>>> >>>>>> If anyone has any ideas they would be really appreciated. I feel >>>>>> like I've hit a brick wall :/ >>>>>> >>>>>> Thanks >>>>>> >>>>>> James >>>>>> >>>>> >>>>> >>>> >>> >> > >
