Hi Robert, Robert Metzger wrote:
> Hi, > > I'm a developer at the Apache Flink (incubating) project. I'm trying to > pass the version of a dependency using a property from the command line to > the binary build. But it seems that the version I've set is not respected > everywhere. > > > We have the following setup: > - flink-parent > - flink-core > - ... more modules ... > - flink-yarn > - flink-dist > > The "flink-yarn" module (and others) have dependencies to Apache Hadoop. > We have a property (called hadoop.version) to have one place for setting > the Hadoop version. The default Hadoop version 2.2.0. > > The "flink-dist" module is creating the binary release for Flink. It is > using the maven-assembly-plugin for that. All jars we depend on are placed > in the "lib" directory. > The "flink-dist" module has most of our modules (also the flink-yarn) > module as a dependency so that we can use the modules in the assembly. > > > > What I'm trying to do is building Flink for a different Hadoop version, > say "2.2.0-cdh5.0.0-beta-2". Therefore, I'm passing this version as > "-Dhadoop.version=2.2.0-cdh5.0.0-beta-2". > > However, some files from Hadoop "2.2.0" end up in the "lib" directory. > > mvn dependency:tree reveals for "flink-dist": > > [INFO] +- org.apache.flink:flink-yarn:jar:0.8-incubating-SNAPSHOT:compile > [INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile > [INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile > [INFO] | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile > > however, for "flink-yarn" the versions are correct: > > [INFO] +- > org.apache.hadoop:hadoop-yarn-client:jar:2.2.0-cdh5.0.0-beta-2:compile > [INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile > [INFO] | +- commons-lang:commons-lang:jar:2.6:compile > > Here is the full dependency:tree output: > https://gist.github.com/rmetzger/70c5f35d4cfd06a91169 > > > How can I create builds for different Hadoop versions using Maven? General rule: - don't use a version tag for direct dependencies - use a common parent for all modules with - a dependency management section for any dependency you want to control - use properties for these versions These properties can be overwritten from command line or setting the property again explicitly in a module. Cheers, Jörg --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
