Re: Oozie Job + Version Control Tool

[Peter Cseh]

I'd like to pour this conversation into
https://issues.apache.org/jira/browse/OOZIE-2876 and it's subtasks. It
looks like those action types are trying to solve these issues.
Please share your thoughts on this approach there.
Thanks!

gp


On Mon, Oct 31, 2016 at 8:12 AM, goun na  wrote:

> maven-assembly-plugin is good to use separate packaging. Thanks. :)
>
> 2016-10-20 15:05 GMT+09:00 Per Ullberg :
>
> > Here's a pom, but I don't think it will tell you much.
> >
> >
> > 
> > http://maven.apache.org/POM/4.0.0; xmlns:xsi="
> > http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="
> > http://maven.apache.org/POM/4.0.0
> > http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> > 
> > hbase-to-hdfs-parent
> > com.klarna.datavault
> > 4.1.1-SNAPSHOT
> > 
> > 4.0.0
> >
> > hbase-to-hdfs
> >
> > 
> >
> > ...
> >
> > 
> > com.github.spullara.mustache.java
> > compiler
> > 0.8.18
> > 
> >
> > 
> > joda-time
> > joda-time
> > 2.8
> > test
> > 
> > 
> > com.klarna
> > hiverunner
> > 2.6.0
> > test
> > 
> > 
> > junit
> > junit
> > 
> > 
> > org.apache.calcite
> > calcite-avatica
> > 
> > 
> > org.apache.calcite
> > calcite-core
> > 
> > 
> > 
> > 
> >
> >
> > 
> > 
> > 
> > src/main/resources
> > true
> > 
> > 
> > src/main/templates
> > true
> > 
> > 
> >
> > 
> > 
> > org.apache.maven.plugins
> > maven-assembly-plugin
> > 2.4
> > 
> > assembly-descriptor.xml
> > 
> > 
> > 
> > package
> > 
> > single
> > 
> > 
> > 
> > 
> > 
> > 
> >
> > 
> >
> >
> > We use mustache a lot to materialize coordinators, workflows and hql,
> both
> > buildtime and runtime. A nice thing we came up with is that some of our
> > workflows have sub-workflows that gets materialized by the main workflow
> at
> > runtime. That way, different sub workflows may be generated dependent on
> a
> > runtime configuration. As an example we have sub workflows that load data
> > from either to postgres or to kafka. These two ways of loading are
> > described in two separate sub-workflow templates and are materialized at
> > runtime dependent on if the user configured a postgres or a kafka load.
> >
> > This way we can build workflows that can be more reactive to the current
> > state of the cluster and configuration.
> >
> > Here's an oozie snippet:
> >
> > 
> > 
> > 
> > 
> > 
> > com.klarna.datavault.load.materialize.
> > MaterializeLoadSubWorkflowMain
> > 
> > -o
> > ${wf:appPath()}/load/load_sub_workflow-${wf:id()}.xml
> > 
> > -k
> > ${wf:appPath()}/load/kafka.properties
> > -w
> > ${workflowConfigPath}
> > -d
> > ${TARGET_DB_LOCATION}
> > -s
> > ${wf:appPath()}/load/hql-delta-transform-${wf:id()}.
> hql
> > -n
> > ${TARGET_DB_NAME}
> > -c
> > ${LOAD_DIFF_DB_LOCATION}
> > ${HIVE_SITE_XML}
> > 
> > 
> > 
> > 
> > 
> >
> >
> > 
> > 
> > ${wf:appPath()}/load/load_sub_workflow-${wf:
> > id()}.xml
> > 
> > 
> > 
> > jobName
> > ${jobName}-load
> > 
> > 
> > parentWorkflowAppPath
> > ${wf:appPath()}
> > 
> > 
> > kafkaConfigFilePath
> > ${wf:appPath()}/load/kafka.properties
> > 
> > 
> > hqlDeltaTransformationPath
> > load/hql-delta-transform-${wf:id()}.hql
> > 
> > 
> > 
> > 
> > 
> > 
> >
> >
> >
> > Sorry if I'm blabbing
> > /Pelle
> >
> > On Thu, Oct 20, 2016 at 3:52 AM, goun na  wrote:
> >
> > > Per Ullberg, a snippet of pom.xml would help us. :)
> > > Thanks,
> > >
> > > 2016-10-20 3:36 GMT+09:00 Per Ullberg :
> > >
> > > > @goun na: we keep one coordinator per (zip|war|jar)
> > > >
> > > > @shiva: I'm happy to share, but it's hard to know what you're in need
> > of.
> > > > Ask and I will try to answer 

Re: OYA! - no more LauncherMappers

[Robert Kanter]

Thanks for getting this in.  I think this is going to greatly improve
things for Oozie going forward, and allow us to address common issues we
couldn't before.

On Fri, May 26, 2017 at 4:23 AM, Peter Bacsko  wrote:

> Finally, it's a milestone :)
>
> By the way, we should summarize what changed and how it affects current
> Oozie workflows (perhaps adding something to the Oozie website makes sense
> too). Right now OYA lacks good backward compatibility, which means that MR
> properties don't have any effect. This will change though.
>
> Another thing that we have to agree on is patching: lots of stuff changed
> inside JavaActionExecutor, LauncherMapper no longer exists, etc. Therefore
> if someone discovers a problem in these classes (or just wants to add an
> improvement), it will not be possible to patch them. I suggest branching
> off from the commit before OYA and then we can start maintaining two
> branches. But this is just an idea.
>
> Peter
>
> On Fri, May 26, 2017 at 12:40 PM, Peter Cseh 
> wrote:
>
> > Hi everyone,
> >
> > OOZIE-1770 - Create Oozie Application Master for YARN is committed to
> > master!
> > I would like to thank the effort of everybody who was involved in the
> > design, the development or provided feedback in Jira or on ReviewBoard.
> >
> > This is a big change in how Oozie works and there is still a lot to do:
> you
> > can check out OOZIE-2889 for details.
> >
> > Thanks again everyone!
> > gp
> >
> >
> > --
> > Peter Cseh
> > Software Engineer
> > 
> >
>

Re: SLA alerts on running actions

[Purshotam Shah]

I think that was a bug in oozie and it got fixed in Oozie-4.3.0.Which version 
of oozie are you using?


On Friday, May 26, 2017, 5:06:53 AM PDT, Artem Ervits  
wrote:You can access SLA information via JMS or REST in case you need that info
prior to job completion. Also check the following page, there is some
tuning available to make processing of SLA events faster
https://oozie.apache.org/docs/4.0.0/AG_Install.html#Notifications_Configuration


On May 24, 2017 11:14 AM, "Suresh V"  wrote:

> Hello Oozie users - we want Oozie to send an email notification when a
> workflow action is running more than a set duration.
>
> We experimented with the SLA feature however the message is sent only after
> the action is completed, not when the action is in running state and the
> duration is passed.
>
> Is there a way to achieve notification while action is running?
>
> Thnak you
> Suresh.
>