It works. The current master should work for all programs started out of Eclipse/IntelliJ (testted by Aljoscha and me) and exposes correct dependencies through maven.
Please post if you have further trouble. On Wed, Oct 1, 2014 at 6:25 PM, Stephan Ewen <[email protected]> wrote: > Just had an idea: > > We can let the IDEs export guava from the "flink-shaded" project and have > the maven builds use the autogenerated dependency-reduced-pom to hide the > original guava. > > Let me try this... > > On Wed, Oct 1, 2014 at 6:10 PM, Stephan Ewen <[email protected]> wrote: > >> Sorry for the hassle, I know this whole dependency thing is tedious, but >> I think it is important that we get our guava out of the way. Otherwise >> anyone that uses Flink will be in the trouble that we are with hadoop (that >> did not shade its guava). >> >> I had the same issue. In a prior version, we had guava as provided in the >> parent pom. >> >> The hadoop dependency (that needs guava) thought guava will be provided >> and did not add it, ultimately leading to failures. We need to make sure >> that a dependency that depends on guava adds its guava version, while our >> own code must not add it (our code must only assume its presence - it is >> rewritten to the shaded version anyways). >> >> I am not sure how to simplify that, I tried various things for many many >> hours and I did not find a better approach. >> >> The only alternative I could come up with is shading guava in each of our >> projects separately, which means that the guava classes will be added to >> that project's jar file, in a different name space. Since we have 10+ >> projects depending on guava, we get guava 10 times into our jars in total. >> Not a clean solution as well (although it works, as the class loaders >> ignore duplicate classes. >> >> What would be cool is if we had a maven option to "unprovide" guava. The >> we could provide in the parent pom, and unprovide for the hadoop dependency. >> >> >> >> >> >> On Wed, Oct 1, 2014 at 5:43 PM, Márton Balassi <[email protected]> >> wrote: >> >>> I've just updated the streaming fork with Stefan's recent commit >>> encapsulating this issue, namely: >>> >>> https://github.com/mbalassi/incubator-flink/commit/949699dbfe17b62352413769635aed3aaff56100 >>> >>> It solves the problem for the streaming-core project, but running the >>> batch >>> WordCount example in Eclipse still produces the following: >>> >>> Executing WordCount example with built-in default data. >>> Provide parameters to read input data from a file. >>> Usage: WordCount <text path> <result path> >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> com/google/common/base/Preconditions >>> at >>> >>> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper.<init>(UserCodeObjectWrapper.java:40) >>> at >>> >>> org.apache.flink.api.common.operators.base.GenericDataSourceBase.<init>(GenericDataSourceBase.java:58) >>> at >>> >>> org.apache.flink.api.java.operators.DataSource.translateToDataFlow(DataSource.java:75) >>> at >>> >>> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:82) >>> at >>> >>> org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:117) >>> at >>> >>> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:85) >>> at >>> >>> org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:117) >>> at >>> >>> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:85) >>> at >>> >>> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:60) >>> at >>> >>> org.apache.flink.api.java.operators.OperatorTranslation.translateToPlan(OperatorTranslation.java:48) >>> at >>> >>> org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:650) >>> at >>> >>> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:48) >>> at >>> >>> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82) >>> Caused by: java.lang.ClassNotFoundException: >>> com.google.common.base.Preconditions >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>> ... 13 more >>> >>> Of course one approach is to move >>> >>> <groupId>com.google.guava</groupId> >>> <artifactId>guava</artifactId> >>> <version>${guava.version}</version> >>> <scope>provided</scope> >>> </dependency> >>> >>> towards the flink-parent pom, but this ultimately works against shading >>> and >>> the hadoop 2.5 version is going to fail as it does here: >>> >>> https://travis-ci.org/mbalassi/incubator-flink/jobs/36780802 >>> >>> Any suggestions? :) >>> >>> On Wed, Oct 1, 2014 at 9:40 AM, Gyula Fora <[email protected]> wrote: >>> >>> > That worked, thanks. >>> > >>> > On 30 Sep 2014, at 23:12, Stephan Ewen <[email protected]> wrote: >>> > >>> > > Yes, I wanted to write a few pointers about such issues. >>> > > >>> > > We recently shaded the guava dependency, meaning that we have a >>> custom >>> > > version of guava where all classes reside in >>> > > "org.apache.flink.shaded.com.google" and maven rewrites all our guava >>> > > references to that namespace. >>> > > >>> > > The reason is that guava is a frequently used library and versions >>> are >>> > not >>> > > all compatible. For example, hadoop 2.5 uses guava 11 and fails if we >>> > bring >>> > > guava 17 into the classpath. The shading causes our guava version to >>> not >>> > > conflict, because our shaded version's classes reside different >>> > namespace. >>> > > >>> > > Since Eclipse does not interpret the shading (it only happens in the >>> > maven >>> > > package phase) you need a regular dependency to guava that is not >>> > exported >>> > > ("provided" scope) >>> > > >>> > > Try adding this to the project's pom, it will tell maven to compile >>> and >>> > > individually run with the guava lib, but not package it. >>> > > >>> > > <dependency> >>> > > <groupId>com.google.guava</groupId> >>> > > <artifactId>guava</artifactId> >>> > > <version>${guava.version}</version> >>> > > <scope>provided</scope> >>> > > </dependency> >>> > > >>> > > >>> > > >>> > > On Tue, Sep 30, 2014 at 11:02 PM, Gyula Fóra <[email protected]> >>> wrote: >>> > > >>> > >> Hey, >>> > >> >>> > >> We have pulled the dependency rework from the apache master and now >>> we >>> > >> cannot get our tests to run in eclipse. With maven from the command >>> line >>> > >> and also with travis it works perfectly but when I try to run tests >>> that >>> > >> access the the Configuration class we get the following exception: >>> > >> >>> > >> java.lang.NoClassDefFoundError: com/google/common/io/BaseEncoding >>> > >> at >>> > >> >>> > >> >>> > >>> org.apache.flink.configuration.Configuration.setBytes(Configuration.java:358) >>> > >> at >>> > >> ..... >>> > >> some stuff here >>> > >> ..... >>> > >> >>> > >> Caused by: java.lang.ClassNotFoundException: >>> > >> com.google.common.io.BaseEncoding >>> > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>> > >> at java.security.AccessController.doPrivileged(Native Method) >>> > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>> > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>> > >> ... 32 more >>> > >> >>> > >> I literally tried everything to make it work, deleted all maven >>> files, >>> > >> reinstalled eclipse etc, but still no luck. Do you experience the >>> same >>> > >> issue when running the streaming-core tests in eclipse? >>> > >> >>> > >> Regards, >>> > >> Gyula >>> > >> >>> > >>> > >>> >> >> >
