> On 2012-02-02 07:59:13, Mohammad Islam wrote: > > So it is WIP. right? > > > > So we are now bundling the multiple hadoop jars. As the new version of > > hadoop comes, we will need add a new pom file and associated changes in > > other pom files. > > Moreover (in some cases) to support a newly released version we might need > > to release new oozie version with only new hadoop packages. > > In addition, oozie tarball size will increase substantially. > > > > I think we might need to be little cautious on this. > > > > Alejandro Abdelnur wrote: > Mohammed, > > The patch is not WIP, is ready for commit. > > I assume you mean that will be evolving, adding more Hadoop versions as > they come, if that is what you meant you are correct. > > When a new version of Hadoop comes and we wan to support it in Oozie, we > would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in > Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it > other than setting the desired version from the command line when running > Maven '-Dhadoop.version=#####'. > > As we stop supporting builds/tests for a particular version of Hadoop we > would remove the corresponding hadoop-client/hadoop-test modules from > hadooplibs. > > Apache projects are Open Source projects and the official distribution of > release should be a source tarball. Note that in the source form this does > not increase the size of the TAR in a considerable way (just a few KBs for > the 2 new POMs). > > For convenience, many projects also include binary distributions. You are > correct that the more Hadoop versions we have hadooplibs the bigger will be > the size of the binary distribution. We have different options for this: > > 1* The binary distribution does not include/bundle/produce hadooplibs. > The size of Oozie TAR remain will remain constant. This is what we have > today, the user is responsible for harvesting the required Hadoop JARs and > can use the hadooplib poms as a guidance. > 2* The binary distribution would be multiple TARs, one for Oozie proper > and one for each hadooplib. The size of Oozie TAR will remain constant, the > user will have to download the corresponding hadooplib tar. > 3* Bundle all hadooplibs as part of the binary distribution. (As you've > mentioned) The size of the TAR will increase. > > Doing #1 the binary TAR is not different from today's. > Doing #2 the binary TAR is not different from today's, you have > additional Oozie hadooplib tars. > Doing #3 the binary TAR will be bigger from today's. > > We can do #1, thus our concerns are addressed and our build easily > handles multiple versions of Hadoop. > > Thoughts? > > Thanks > > Alejandro > > > > >
I prefer both option #1 and option #. But Option #2 will be better for easy/quick installation. So pls go ahead with option #2. One question: do we need to add pom for every new hadoop (major/minor) release (although there is no changes in the jar files? - Mohammad ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3726/#review4764 ----------------------------------------------------------- On 2012-02-01 20:18:21, Alejandro Abdelnur wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/3726/ > ----------------------------------------------------------- > > (Updated 2012-02-01 20:18:21) > > > Review request for oozie. > > > Summary > ------- > > The current mechanism allow to easily package/use new versions of hadoop > without complicating Oozie's POMs. > > mr1 & mr2 profiles are gone from oozie, now it refers to one of the versions > in hadooplibs. > > New maven modules under hadooplib define the hadoop-client/hadoop-test POMs > for different hadoop versions. > > Note that because of HADOOP-8009, hadoop will start providing a hadoop-client > artifact (even for already released versions), still we'll need the > corresponding hadooplibs module to be able to use the assembly (as it is done > as part of this patch) to pull into the oozie distro the JARs for the > supported/tested versions of Hadoop required for the client side. > > Note that this can be used tosimplify the logic of addtowar.sh that won't > have to be aware of the JARs deps of different versions of Hadoop or of > hadoop JARs at all. > > > This addresses bug OOZIE-667. > https://issues.apache.org/jira/browse/OOZIE-667 > > > Diffs > ----- > > trunk/core/pom.xml 1239282 > trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282 > trunk/docs/src/site/twiki/AG_Install.twiki 1239282 > trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282 > trunk/examples/pom.xml 1239282 > trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION > trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION > trunk/hadooplibs/pom.xml PRE-CREATION > trunk/pom.xml 1239282 > trunk/sharelib/pom.xml 1239282 > trunk/sharelib/streaming/pom.xml 1239282 > trunk/src/main/assemblies/distro.xml 1239282 > trunk/src/main/assemblies/hadooplib.xml PRE-CREATION > trunk/src/main/assemblies/partial-sharelib.xml 1239282 > trunk/webapp/pom.xml 1239282 > > Diff: https://reviews.apache.org/r/3726/diff > > > Testing > ------- > > Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here but > that is still work in progress for the mr2 integration) > > > Thanks, > > Alejandro > >
