> On 2012-02-02 07:59:13, Mohammad Islam wrote: > > So it is WIP. right? > > > > So we are now bundling the multiple hadoop jars. As the new version of > > hadoop comes, we will need add a new pom file and associated changes in > > other pom files. > > Moreover (in some cases) to support a newly released version we might need > > to release new oozie version with only new hadoop packages. > > In addition, oozie tarball size will increase substantially. > > > > I think we might need to be little cautious on this. > >
Mohammed, The patch is not WIP, is ready for commit. I assume you mean that will be evolving, adding more Hadoop versions as they come, if that is what you meant you are correct. When a new version of Hadoop comes and we wan to support it in Oozie, we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other than setting the desired version from the command line when running Maven '-Dhadoop.version=#####'. As we stop supporting builds/tests for a particular version of Hadoop we would remove the corresponding hadoop-client/hadoop-test modules from hadooplibs. Apache projects are Open Source projects and the official distribution of release should be a source tarball. Note that in the source form this does not increase the size of the TAR in a considerable way (just a few KBs for the 2 new POMs). For convenience, many projects also include binary distributions. You are correct that the more Hadoop versions we have hadooplibs the bigger will be the size of the binary distribution. We have different options for this: 1* The binary distribution does not include/bundle/produce hadooplibs. The size of Oozie TAR remain will remain constant. This is what we have today, the user is responsible for harvesting the required Hadoop JARs and can use the hadooplib poms as a guidance. 2* The binary distribution would be multiple TARs, one for Oozie proper and one for each hadooplib. The size of Oozie TAR will remain constant, the user will have to download the corresponding hadooplib tar. 3* Bundle all hadooplibs as part of the binary distribution. (As you've mentioned) The size of the TAR will increase. Doing #1 the binary TAR is not different from today's. Doing #2 the binary TAR is not different from today's, you have additional Oozie hadooplib tars. Doing #3 the binary TAR will be bigger from today's. We can do #1, thus our concerns are addressed and our build easily handles multiple versions of Hadoop. Thoughts? Thanks Alejandro - Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3726/#review4764 ----------------------------------------------------------- On 2012-02-01 20:18:21, Alejandro Abdelnur wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/3726/ > ----------------------------------------------------------- > > (Updated 2012-02-01 20:18:21) > > > Review request for oozie. > > > Summary > ------- > > The current mechanism allow to easily package/use new versions of hadoop > without complicating Oozie's POMs. > > mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions > in hadooplibs. > > New maven modules under hadooplib define the hadoop-client/hadoop-test POMs > for different hadoop versions. > > Note that because of HADOOP-8009, hadoop will start providing a hadoop-client > artifact (even for already released versions), still we'll need the > corresponding hadooplibs module to be able to use the assembly (as it is done > as part of this patch) to pull into the oozie distro the JARs for the > supported/tested versions of Hadoop required for the client side. > > Note that this can be used tosimplify the logic of addtowar.sh that won't > have to be aware of the JARs deps of different versions of Hadoop or of > hadoop JARs at all. > > > This addresses bug OOZIE-667. > https://issues.apache.org/jira/browse/OOZIE-667 > > > Diffs > ----- > > trunk/core/pom.xml 1239282 > trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282 > trunk/docs/src/site/twiki/AG_Install.twiki 1239282 > trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282 > trunk/examples/pom.xml 1239282 > trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION > trunk/hadooplibs/pom.xml PRE-CREATION > trunk/pom.xml 1239282 > trunk/sharelib/pom.xml 1239282 > trunk/sharelib/streaming/pom.xml 1239282 > trunk/src/main/assemblies/distro.xml 1239282 > trunk/src/main/assemblies/hadooplib.xml PRE-CREATION > trunk/src/main/assemblies/partial-sharelib.xml 1239282 > trunk/webapp/pom.xml 1239282 > > Diff: https://reviews.apache.org/r/3726/diff > > > Testing > ------- > > Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but > that is still work in progress for the mr2 integration) > > > Thanks, > > Alejandro > >
