Just had an idea: We can let the IDEs export guava from the "flink-shaded" project and have the maven builds use the autogenerated dependency-reduced-pom to hide the original guava.
Let me try this... On Wed, Oct 1, 2014 at 6:10 PM, Stephan Ewen <[email protected]> wrote: > Sorry for the hassle, I know this whole dependency thing is tedious, but I > think it is important that we get our guava out of the way. Otherwise > anyone that uses Flink will be in the trouble that we are with hadoop (that > did not shade its guava). > > I had the same issue. In a prior version, we had guava as provided in the > parent pom. > > The hadoop dependency (that needs guava) thought guava will be provided > and did not add it, ultimately leading to failures. We need to make sure > that a dependency that depends on guava adds its guava version, while our > own code must not add it (our code must only assume its presence - it is > rewritten to the shaded version anyways). > > I am not sure how to simplify that, I tried various things for many many > hours and I did not find a better approach. > > The only alternative I could come up with is shading guava in each of our > projects separately, which means that the guava classes will be added to > that project's jar file, in a different name space. Since we have 10+ > projects depending on guava, we get guava 10 times into our jars in total. > Not a clean solution as well (although it works, as the class loaders > ignore duplicate classes. > > What would be cool is if we had a maven option to "unprovide" guava. The > we could provide in the parent pom, and unprovide for the hadoop dependency. > > > > > > On Wed, Oct 1, 2014 at 5:43 PM, Márton Balassi <[email protected]> > wrote: > >> I've just updated the streaming fork with Stefan's recent commit >> encapsulating this issue, namely: >> >> https://github.com/mbalassi/incubator-flink/commit/949699dbfe17b62352413769635aed3aaff56100 >> >> It solves the problem for the streaming-core project, but running the >> batch >> WordCount example in Eclipse still produces the following: >> >> Executing WordCount example with built-in default data. >> Provide parameters to read input data from a file. >> Usage: WordCount <text path> <result path> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> com/google/common/base/Preconditions >> at >> >> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper.<init>(UserCodeObjectWrapper.java:40) >> at >> >> org.apache.flink.api.common.operators.base.GenericDataSourceBase.<init>(GenericDataSourceBase.java:58) >> at >> >> org.apache.flink.api.java.operators.DataSource.translateToDataFlow(DataSource.java:75) >> at >> >> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:82) >> at >> >> org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:117) >> at >> >> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:85) >> at >> >> org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:117) >> at >> >> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:85) >> at >> >> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:60) >> at >> >> org.apache.flink.api.java.operators.OperatorTranslation.translateToPlan(OperatorTranslation.java:48) >> at >> >> org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:650) >> at >> >> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:48) >> at >> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82) >> Caused by: java.lang.ClassNotFoundException: >> com.google.common.base.Preconditions >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> ... 13 more >> >> Of course one approach is to move >> >> <groupId>com.google.guava</groupId> >> <artifactId>guava</artifactId> >> <version>${guava.version}</version> >> <scope>provided</scope> >> </dependency> >> >> towards the flink-parent pom, but this ultimately works against shading >> and >> the hadoop 2.5 version is going to fail as it does here: >> >> https://travis-ci.org/mbalassi/incubator-flink/jobs/36780802 >> >> Any suggestions? :) >> >> On Wed, Oct 1, 2014 at 9:40 AM, Gyula Fora <[email protected]> wrote: >> >> > That worked, thanks. >> > >> > On 30 Sep 2014, at 23:12, Stephan Ewen <[email protected]> wrote: >> > >> > > Yes, I wanted to write a few pointers about such issues. >> > > >> > > We recently shaded the guava dependency, meaning that we have a custom >> > > version of guava where all classes reside in >> > > "org.apache.flink.shaded.com.google" and maven rewrites all our guava >> > > references to that namespace. >> > > >> > > The reason is that guava is a frequently used library and versions are >> > not >> > > all compatible. For example, hadoop 2.5 uses guava 11 and fails if we >> > bring >> > > guava 17 into the classpath. The shading causes our guava version to >> not >> > > conflict, because our shaded version's classes reside different >> > namespace. >> > > >> > > Since Eclipse does not interpret the shading (it only happens in the >> > maven >> > > package phase) you need a regular dependency to guava that is not >> > exported >> > > ("provided" scope) >> > > >> > > Try adding this to the project's pom, it will tell maven to compile >> and >> > > individually run with the guava lib, but not package it. >> > > >> > > <dependency> >> > > <groupId>com.google.guava</groupId> >> > > <artifactId>guava</artifactId> >> > > <version>${guava.version}</version> >> > > <scope>provided</scope> >> > > </dependency> >> > > >> > > >> > > >> > > On Tue, Sep 30, 2014 at 11:02 PM, Gyula Fóra <[email protected]> >> wrote: >> > > >> > >> Hey, >> > >> >> > >> We have pulled the dependency rework from the apache master and now >> we >> > >> cannot get our tests to run in eclipse. With maven from the command >> line >> > >> and also with travis it works perfectly but when I try to run tests >> that >> > >> access the the Configuration class we get the following exception: >> > >> >> > >> java.lang.NoClassDefFoundError: com/google/common/io/BaseEncoding >> > >> at >> > >> >> > >> >> > >> org.apache.flink.configuration.Configuration.setBytes(Configuration.java:358) >> > >> at >> > >> ..... >> > >> some stuff here >> > >> ..... >> > >> >> > >> Caused by: java.lang.ClassNotFoundException: >> > >> com.google.common.io.BaseEncoding >> > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> > >> at java.security.AccessController.doPrivileged(Native Method) >> > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> > >> ... 32 more >> > >> >> > >> I literally tried everything to make it work, deleted all maven >> files, >> > >> reinstalled eclipse etc, but still no luck. Do you experience the >> same >> > >> issue when running the streaming-core tests in eclipse? >> > >> >> > >> Regards, >> > >> Gyula >> > >> >> > >> > >> > >
