[
https://issues.apache.org/jira/browse/OOZIE-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199261#comment-13199261
]
[email protected] commented on OOZIE-667:
-----------------------------------------------------
bq. On 2012-02-02 07:59:13, Mohammad Islam wrote:
bq. > So it is WIP. right?
bq. >
bq. > So we are now bundling the multiple hadoop jars. As the new version of
hadoop comes, we will need add a new pom file and associated changes in other
pom files.
bq. > Moreover (in some cases) to support a newly released version we might
need to release new oozie version with only new hadoop packages.
bq. > In addition, oozie tarball size will increase substantially.
bq. >
bq. > I think we might need to be little cautious on this.
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammed,
bq.
bq. The patch is not WIP, is ready for commit.
bq.
bq. I assume you mean that will be evolving, adding more Hadoop versions
as they come, if that is what you meant you are correct.
bq.
bq. When a new version of Hadoop comes and we wan to support it in Oozie,
we would add 1 new set of hadoop-client/hadoop-test for that Hadoop version in
Oozie's hadooplibs, correct. No other changes in the Oozie POMs to use it other
than setting the desired version from the command line when running Maven
'-Dhadoop.version=#####'.
bq.
bq. As we stop supporting builds/tests for a particular version of Hadoop
we would remove the corresponding hadoop-client/hadoop-test modules from
hadooplibs.
bq.
bq. Apache projects are Open Source projects and the official distribution
of release should be a source tarball. Note that in the source form this does
not increase the size of the TAR in a considerable way (just a few KBs for the
2 new POMs).
bq.
bq. For convenience, many projects also include binary distributions. You
are correct that the more Hadoop versions we have hadooplibs the bigger will be
the size of the binary distribution. We have different options for this:
bq.
bq. 1* The binary distribution does not include/bundle/produce hadooplibs.
The size of Oozie TAR remain will remain constant. This is what we have today,
the user is responsible for harvesting the required Hadoop JARs and can use the
hadooplib poms as a guidance.
bq. 2* The binary distribution would be multiple TARs, one for Oozie
proper and one for each hadooplib. The size of Oozie TAR will remain constant,
the user will have to download the corresponding hadooplib tar.
bq. 3* Bundle all hadooplibs as part of the binary distribution. (As
you've mentioned) The size of the TAR will increase.
bq.
bq. Doing #1 the binary TAR is not different from today's.
bq. Doing #2 the binary TAR is not different from today's, you have
additional Oozie hadooplib tars.
bq. Doing #3 the binary TAR will be bigger from today's.
bq.
bq. We can do #1, thus our concerns are addressed and our build easily
handles multiple versions of Hadoop.
bq.
bq. Thoughts?
bq.
bq. Thanks
bq.
bq. Alejandro
bq.
bq.
bq.
bq.
bq.
I prefer both option #1 and option #.
But Option #2 will be better for easy/quick installation.
So pls go ahead with option #2.
One question: do we need to add pom for every new hadoop (major/minor) release
(although there is no changes in the jar files?
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3726/#review4764
-----------------------------------------------------------
On 2012-02-01 20:18:21, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3726/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-01 20:18:21)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The current mechanism allow to easily package/use new versions of hadoop
without complicating Oozie's POMs.
bq.
bq. mr1 & mr2 profiles are gone from oozie, now it refers to one of the
versions in hadooplibs.
bq.
bq. New maven modules under hadooplib define the hadoop-client/hadoop-test
POMs for different hadoop versions.
bq.
bq. Note that because of HADOOP-8009, hadoop will start providing a
hadoop-client artifact (even for already released versions), still we'll need
the corresponding hadooplibs module to be able to use the assembly (as it is
done as part of this patch) to pull into the oozie distro the JARs for the
supported/tested versions of Hadoop required for the client side.
bq.
bq. Note that this can be used tosimplify the logic of addtowar.sh that won't
have to be aware of the JARs deps of different versions of Hadoop or of hadoop
JARs at all.
bq.
bq.
bq. This addresses bug OOZIE-667.
bq. https://issues.apache.org/jira/browse/OOZIE-667
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/pom.xml 1239282
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1239282
bq. trunk/docs/src/site/twiki/AG_Install.twiki 1239282
bq. trunk/docs/src/site/twiki/DG_QuickStart.twiki 1239282
bq. trunk/examples/pom.xml 1239282
bq. trunk/hadooplibs/hadoop-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_23_1/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-0_24_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/hadoop-test-1_0_0/pom.xml PRE-CREATION
bq. trunk/hadooplibs/pom.xml PRE-CREATION
bq. trunk/pom.xml 1239282
bq. trunk/sharelib/pom.xml 1239282
bq. trunk/sharelib/streaming/pom.xml 1239282
bq. trunk/src/main/assemblies/distro.xml 1239282
bq. trunk/src/main/assemblies/hadooplib.xml PRE-CREATION
bq. trunk/src/main/assemblies/partial-sharelib.xml 1239282
bq. trunk/webapp/pom.xml 1239282
bq.
bq. Diff: https://reviews.apache.org/r/3726/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with 1.0.0 and 0.24.0-SNAPSHOTs (several testcases failures here
but that is still work in progress for the mr2 integration)
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Change the way Oozie brings in Hadoop JARs into the build
> ---------------------------------------------------------
>
> Key: OOZIE-667
> URL: https://issues.apache.org/jira/browse/OOZIE-667
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently we are using profiles with several exclussions and they are through
> out the POMs.
> A cleaner way, similar to sharelibs, is to have hadooplibs where there is a
> lib per version of Hadoop.
> Each Hadooplib would have the correct JARs for that version. These hadooplibs
> would be bundled with the release. Then it would be much easier for
> oozie-setup.sh just to point at the hadoplib dir required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
