> On 2011-11-22 23:26:39, Mohammad Islam wrote:
> > Overall, it's an important problem.
> >
> > In every product, we could use some sub-dirs for future (such as lib/
> > conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for
> > classes/jars.
> >
> > Alternatively, we can coordinate this with multi-versions. There could be
> > multiple directories like $SHARELIB/pig/default/{lib, conf},
> > $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm
> > leaning towards this way to make it consistent with multi-version support.
> >
> >
>
> Alejandro Abdelnur wrote:
> Yes, could have lib/ & conf/. However this is not a requirement at the
> moment, and the change would be transparent when we need it. Like moving from
> a single lib/ to a per-action one, users apps won't be affected.
>
> Adding multi version support is a good idea. It would also require
> changes in the action XML to specify the version.
>
> Both could be future enhancements, but we should think more the use cases
> and requirements. The current patch addresses an immediate need that could be
> augmented later.
>
> Alejandro Abdelnur wrote:
> I'm not against that, actually I like it. But I do that as a follow up
> feature. Note that, similar from the current single lib/ to multiple dirs
> (one per action) we are not breaking applications, it is transparent, the
> case will be when we ad lib/ and conf/ and versions. BTW, handling version
> requires some thought on how the user will specify the version he/she wants
> to use (in the schema URI, a config property, a new element; this has to be
> thought). Because of this I'd like to do things incrementally, the current
> patch now and a follow up JIRA to discuss your proposal.
>
> Thanks.
>
> Mohammad Islam wrote:
> I think the "use_version" support is considered here.
> Long time back, Virag had a pending patch :
> https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
> Please review this patch.
>
> We had a long discussion on this design. Main proposal is captured here
> (some of your comments lost in the old JIRA system)
> https://issues.apache.org/jira/browse/OOZIE-555
>
>
>
>
> Alejandro Abdelnur wrote:
> Mmmh (regarding my lost comments).
>
> OK, the idea is similar, the main difference with what I'm proposing is
> that in Virag's patch it requires a new WF schema and that the library paths
> are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes
> that make difficult to see what is a change and what is not)
>
> To achieve similar functionality on top of OOZIE-610 we could:
>
> * Change the sharelib path to contain action library versions, ie:
>
> share/lib/streaming/0.20.204.0/
> share/lib/pig/0.8.0/
> share/lib/pig/0.9.0/
>
> * Have an oozie-site configuration, per action that defines the stable
> version of the library, ie:
>
> oozie.action.stable.library.version.streaming=0.20.204.0
> oozie.action.stable.library.version.pig=0.8.0
>
> * The ActionExecutor getShareLibPostFix() method would have the following
> logic:
>
> getShareLibPostFix(...) {
>
> String version = action.config.get("oozie.action.library.version");
> if (version == null) {
> version = oozie.config.get("oozie.action.stable.library.version");
> }
> if (version != null) {
> //construct the library path using the specified version
> }
> }
>
> This approach does not require changes in the WF schema, instead uses a
> configuration value in case you want a different version than the stable, if
> not it just works for existing WF apps.
>
>
>
> If we are OK with this approach I'll file a JIRA with it.
>
>
> Mohammad Islam wrote:
> In addition to your proposal, we want to give the user the flexibility to
> choose the version *optionally*. User can specify that in wf.xml using
> use-version. Basically the effective version will be user-specified version ,
> if it is defined in wf.xml. Otherwise, the default one that you mentioned
> will be the effective version.
>
> there is already a JIRA on this:
> https://issues.apache.org/jira/browse/OOZIE-555
> You can add your comments there.
>
> Alejandro Abdelnur wrote:
> In my proposal, if the user sets the configuration property
> 'oozie.action.library.version' to any value, that version will be used. If
> the user does not set it the default stable one will be used.
>
> I'll put my proposal in OOZIE-555
The end-user who writes the workflow.xml should have a way of controlling his
version. For example, one user tested his pig script againist pig 0.8 another
one has used the latest (0.9). We should allow the both users to support that
provide pig 0.8 and pig 0.9 are supported versions. I think it should be
defined per-action level.
Also we had the same design discussion few months back.
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2909/
> -----------------------------------------------------------
>
> (Updated 2011-11-22 21:03:19)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> breaks the Oozie sharelib into one subdir per action type
>
>
> This addresses bug OOZIE-610.
> https://issues.apache.org/jira/browse/OOZIE-610
>
>
> Diffs
> -----
>
> /trunk/core/pom.xml 1205165
> /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
> 1205165
>
> /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> 1205165
>
> /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
> 1205165
>
> /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
> 1205165
> /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
> 1205165
> /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
> /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
> /trunk/examples/pom.xml 1205165
> /trunk/examples/src/main/apps/custom-main/job.properties 1205165
> /trunk/examples/src/main/apps/demo/job.properties 1205165
> /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
> /trunk/examples/src/main/apps/java-main/job.properties 1205165
> /trunk/examples/src/main/apps/pig/job.properties 1205165
> /trunk/examples/src/main/apps/streaming/job.properties 1205165
> /trunk/pom.xml 1205165
> /trunk/sharelib/pig/pom.xml PRE-CREATION
> /trunk/sharelib/pom.xml 1205165
> /trunk/sharelib/streaming/pom.xml PRE-CREATION
> /trunk/src/main/assemblies/distro.xml 1205165
> /trunk/src/main/assemblies/examples.xml 1205165
> /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
> /trunk/src/main/assemblies/sharelib.xml 1205165
>
> Diff: https://reviews.apache.org/r/2909/diff
>
>
> Testing
> -------
>
> built, installed and run pig and streaming examples
>
>
> Thanks,
>
> Alejandro
>
>
