[
https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155694#comment-13155694
]
[email protected] commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. > Overall comments:
bq. > 1. Doing it incrementally will be good idea. But we need to consider the
backward compatibility. For example, currently we are reading pig/*.jar, in
the next oozie release, we decided to support pig/lib/*.jar or
pig/0.8/lib/*.jar or pig/stable/lib/*.jar. In this case, the old setup will not
work.
bq. >
bq. > 2. We wanted to support multiple version too. Virag has the patch. What
is the best way of getting the benefits of both?
bq. >
Backwards compatibility would not be an issue because when you change the
layout of the library you'll change the resolution logic, and for jobs will be
transparent. When you upgrade Oozie you'll have to upgrade the sharelib at the
same time; but WF apps will be unaffected. Furthermore a migration path would
be that now you are only supporting one version, 'stable', and later you'll
support other versions via a configuration in the action XML.
Alternate versions of libraries could be supported by a
'oozie.action.library.version' property in the action configuration, if not set
it would resolve to 'stable', if set it would resolve to the version specified.
This change is not complex, actually it is quite simple. What complicates
things a bit is the packaging of sharelibs.
Again, we could do this incremental (as in a couple of weeks), I just don't
want to delay this patch much more as I have a patch chain (this, Hive, Sqoop,
Alfredo) which takes time and work to keep up to date.
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. > /trunk/docs/src/site/twiki/DG_Examples.twiki, line 95
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59839#file59839line95>
bq. >
bq. > So user level libpath will not be supported?
This is still supported, the examples are not using it anymore, instead they
use the Oozie share lib.
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. >
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java,
line 192
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59838#file59838line192>
bq. >
bq. > why are we not reading the system library path for jars. There could
be some system jars ,that are not any product specific (like pig, hive) but
are commonly used. Also the question of backward compatibility is there.
Is this a real scenario for you or hypothetical? If the latter then I would not
worry as users can still use user set library paths.
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. > /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java,
line 30
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59834#file59834line30>
bq. >
bq. > But I don't see any other code changes in this file.
My bad, thought the prev comment was for the JavaActionExecutor class, will
remove.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3459
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
1205165
bq.
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by
> the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share
> lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is
> dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the
> issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4
> to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a
> single version of each one of them, only one version of antlr-runtime makes
> it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action
> requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action
> classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the
> submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system
> share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs
> for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
