You are asking the right question in the right forum.
I can give you a bit of background that describes why you are having
trouble with Maven.
Maven has a mind of its own.
At the core of its belief system is the basic principle that anyone
should be able to go to your SCM, download some version of your sources
and pom files and tell Maven to build the project and get exactly what
you got when you did this.
Allowing command line arguments to affect the build would violate Maven
core beliefs
Maven will fight any attempt to bend or break its resolve in this regard.
The phrase "Resistence if futile!" comes up in this forum from time to time.
Given that you can't do what you want, here are some ideas that might
give you some better ideas for a solution.
1) Does any of your code actually care at compile time what version of
Hadoop it builds with? If not, you have a run-time issue not really a
build issue.
2) Is you problem really one of testing with different Hadoop versions?
In that case, you can just build your jar with a "typical/generic"Hadoop
<scope>provided</scope> and use that jar with different combinations of
Hadoop versions in various test environments to certify that it works.
2) What if a neutral version of Hadoop was used at compile time but not
included in your jar <scope>provided</scope>and a different one was used
at run-time, would your application run. If so, you could distribute
your application without Hadoop and let the user add their own Hadoop
jars to the classpath at run-time.
3) Are you going to distribute your software with an installer that
would allow you to build various versions of your installer (or add
options to your installer) to install the "right" Hadoop as part of the
installation.
I know that these are not the answers that you wanted or expected but I
hope it gives you some ideas that moves you forward.
On 14/11/2014 9:23 AM, Robert Metzger wrote:
Hi,
I'm a developer at the Apache Flink (incubating) project. I'm trying to
pass the version of a dependency using a property from the command line to
the binary build. But it seems that the version I've set is not respected
everywhere.
We have the following setup:
- flink-parent
- flink-core
- ... more modules ...
- flink-yarn
- flink-dist
The "flink-yarn" module (and others) have dependencies to Apache Hadoop. We
have a property (called hadoop.version) to have one place for setting the
Hadoop version. The default Hadoop version 2.2.0.
The "flink-dist" module is creating the binary release for Flink. It is
using the maven-assembly-plugin for that. All jars we depend on are placed
in the "lib" directory.
The "flink-dist" module has most of our modules (also the flink-yarn)
module as a dependency so that we can use the modules in the assembly.
What I'm trying to do is building Flink for a different Hadoop version, say
"2.2.0-cdh5.0.0-beta-2". Therefore, I'm passing this version as
"-Dhadoop.version=2.2.0-cdh5.0.0-beta-2".
However, some files from Hadoop "2.2.0" end up in the "lib" directory.
mvn dependency:tree reveals for "flink-dist":
[INFO] +- org.apache.flink:flink-yarn:jar:0.8-incubating-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
however, for "flink-yarn" the versions are correct:
[INFO] +-
org.apache.hadoop:hadoop-yarn-client:jar:2.2.0-cdh5.0.0-beta-2:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
Here is the full dependency:tree output:
https://gist.github.com/rmetzger/70c5f35d4cfd06a91169
How can I create builds for different Hadoop versions using Maven?
Regards,
Robert
--
Ron Wheeler
President
Artifact Software Inc
email: [email protected]
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
