[
https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156229#comment-13156229
]
[email protected] commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/
conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for
classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could
be multiple directories like $SHARELIB/pig/default/{lib, conf},
$SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning
towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the
moment, and the change would be transparent when we need it. Like moving from a
single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require
changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use
cases and requirements. The current patch addresses an immediate need that
could be augmented later.
bq.
bq. Alejandro Abdelnur wrote:
bq. I'm not against that, actually I like it. But I do that as a follow up
feature. Note that, similar from the current single lib/ to multiple dirs (one
per action) we are not breaking applications, it is transparent, the case will
be when we ad lib/ and conf/ and versions. BTW, handling version requires some
thought on how the user will specify the version he/she wants to use (in the
schema URI, a config property, a new element; this has to be thought). Because
of this I'd like to do things incrementally, the current patch now and a follow
up JIRA to discuss your proposal.
bq.
bq. Thanks.
bq.
bq. Mohammad Islam wrote:
bq. I think the "use_version" support is considered here.
bq. Long time back, Virag had a pending patch :
https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
bq. Please review this patch.
bq.
bq. We had a long discussion on this design. Main proposal is captured
here (some of your comments lost in the old JIRA system)
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq.
bq.
bq.
bq.
bq. Alejandro Abdelnur wrote:
bq. Mmmh (regarding my lost comments).
bq.
bq. OK, the idea is similar, the main difference with what I'm proposing
is that in Virag's patch it requires a new WF schema and that the library paths
are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes
that make difficult to see what is a change and what is not)
bq.
bq. To achieve similar functionality on top of OOZIE-610 we could:
bq.
bq. * Change the sharelib path to contain action library versions, ie:
bq.
bq. share/lib/streaming/0.20.204.0/
bq. share/lib/pig/0.8.0/
bq. share/lib/pig/0.9.0/
bq.
bq. * Have an oozie-site configuration, per action that defines the stable
version of the library, ie:
bq.
bq. oozie.action.stable.library.version.streaming=0.20.204.0
bq. oozie.action.stable.library.version.pig=0.8.0
bq.
bq. * The ActionExecutor getShareLibPostFix() method would have the
following logic:
bq.
bq. getShareLibPostFix(...) {
bq.
bq. String version = action.config.get("oozie.action.library.version");
bq. if (version == null) {
bq. version = oozie.config.get("oozie.action.stable.library.version");
bq. }
bq. if (version != null) {
bq. //construct the library path using the specified version
bq. }
bq. }
bq.
bq. This approach does not require changes in the WF schema, instead uses
a configuration value in case you want a different version than the stable, if
not it just works for existing WF apps.
bq.
bq.
bq.
bq. If we are OK with this approach I'll file a JIRA with it.
bq.
In addition to your proposal, we want to give the user the flexibility to
choose the version *optionally*. User can specify that in wf.xml using
use-version. Basically the effective version will be user-specified version ,
if it is defined in wf.xml. Otherwise, the default one that you mentioned will
be the effective version.
there is already a JIRA on this:
https://issues.apache.org/jira/browse/OOZIE-555
You can add your comments there.
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by
> the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share
> lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is
> dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the
> issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4
> to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a
> single version of each one of them, only one version of antlr-runtime makes
> it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action
> requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action
> classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the
> submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system
> share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs
> for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
