[
https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155560#comment-13155560
]
[email protected] commented on OOZIE-610:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
Overall, it's an important problem.
In every product, we could use some sub-dirs for future (such as lib/ conf/).
In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
Alternatively, we can coordinate this with multi-versions. There could be
multiple directories like $SHARELIB/pig/default/{lib, conf},
$SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning
towards this way to make it consistent with multi-version support.
/trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7693>
do we need it?
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7695>
so the name will be defined in each action. Or will it be better in the
system configuration in oozie-default(site).xml?
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7697>
Is hard-coding a good approach? Default could be "pig".
- Mohammad
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by
> the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share
> lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is
> dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the
> issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4
> to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a
> single version of each one of them, only one version of antlr-runtime makes
> it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action
> requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action
> classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the
> submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system
> share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs
> for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
