Thanks for creating OOZIE-2876! 2017-05-31 4:26 GMT+09:00 Peter Cseh <gezap...@cloudera.com>:
> I'd like to pour this conversation into > https://issues.apache.org/jira/browse/OOZIE-2876 and it's subtasks. It > looks like those action types are trying to solve these issues. > Please share your thoughts on this approach there. > Thanks! > > gp > > > On Mon, Oct 31, 2016 at 8:12 AM, goun na <gou...@gmail.com> wrote: > > > maven-assembly-plugin is good to use separate packaging. Thanks. :) > > > > 2016-10-20 15:05 GMT+09:00 Per Ullberg <per.ullb...@klarna.com>: > > > > > Here's a pom, but I don't think it will tell you much. > > > > > > > > > <?xml version="1.0" encoding="UTF-8"?> > > > <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi=" > > > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > > > http://maven.apache.org/POM/4.0.0 > > > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > > > <parent> > > > <artifactId>hbase-to-hdfs-parent</artifactId> > > > <groupId>com.klarna.datavault</groupId> > > > <version>4.1.1-SNAPSHOT</version> > > > </parent> > > > <modelVersion>4.0.0</modelVersion> > > > > > > <artifactId>hbase-to-hdfs</artifactId> > > > > > > <dependencies> > > > > > > ... > > > > > > <dependency> > > > <groupId>com.github.spullara.mustache.java</groupId> > > > <artifactId>compiler</artifactId> > > > <version>0.8.18</version> > > > </dependency> > > > > > > <dependency> > > > <groupId>joda-time</groupId> > > > <artifactId>joda-time</artifactId> > > > <version>2.8</version> > > > <scope>test</scope> > > > </dependency> > > > <dependency> > > > <groupId>com.klarna</groupId> > > > <artifactId>hiverunner</artifactId> > > > <version>2.6.0</version> > > > <scope>test</scope> > > > <exclusions> > > > <exclusion> > > > <artifactId>junit</artifactId> > > > <groupId>junit</groupId> > > > </exclusion> > > > <exclusion> > > > <groupId>org.apache.calcite</groupId> > > > <artifactId>calcite-avatica</artifactId> > > > </exclusion> > > > <exclusion> > > > <groupId>org.apache.calcite</groupId> > > > <artifactId>calcite-core</artifactId> > > > </exclusion> > > > </exclusions> > > > </dependency> > > > </dependencies> > > > > > > > > > <build> > > > <resources> > > > <resource> > > > <directory>src/main/resources</directory> > > > <filtering>true</filtering> > > > </resource> > > > <resource> > > > <directory>src/main/templates</directory> > > > <filtering>true</filtering> > > > </resource> > > > </resources> > > > > > > <plugins> > > > <plugin> > > > <groupId>org.apache.maven.plugins</groupId> > > > <artifactId>maven-assembly-plugin</artifactId> > > > <version>2.4</version> > > > <configuration> > > > <descriptor>assembly-descriptor.xml</descriptor> > > > </configuration> > > > <executions> > > > <execution> > > > <phase>package</phase> > > > <goals> > > > <goal>single</goal> > > > </goals> > > > </execution> > > > </executions> > > > </plugin> > > > </plugins> > > > </build> > > > > > > </project> > > > > > > > > > We use mustache a lot to materialize coordinators, workflows and hql, > > both > > > buildtime and runtime. A nice thing we came up with is that some of our > > > workflows have sub-workflows that gets materialized by the main > workflow > > at > > > runtime. That way, different sub workflows may be generated dependent > on > > a > > > runtime configuration. As an example we have sub workflows that load > data > > > from either to postgres or to kafka. These two ways of loading are > > > described in two separate sub-workflow templates and are materialized > at > > > runtime dependent on if the user configured a postgres or a kafka load. > > > > > > This way we can build workflows that can be more reactive to the > current > > > state of the cluster and configuration. > > > > > > Here's an oozie snippet: > > > > > > <action name="materialize_load_sub_workflow" cred="hive_credentials"> > > > <java> > > > <prepare> > > > <mkdir path="${wf:appPath()}/load/"/> > > > </prepare> > > > <main-class>com.klarna.datavault.load.materialize. > > > MaterializeLoadSubWorkflowMain</main-class> > > > <!-- Output location for the materialized workflow XML --> > > > <arg>-o</arg> > > > <arg>${wf:appPath()}/load/load_sub_workflow-${wf:id()}. > xml</arg> > > > <!-- Output location for the materialized kafka.properties > > > file for a Kafka Load --> > > > <arg>-k</arg> > > > <arg>${wf:appPath()}/load/kafka.properties</arg> > > > <arg>-w</arg> > > > <arg>${workflowConfigPath}</arg> > > > <arg>-d</arg> > > > <arg>${TARGET_DB_LOCATION}</arg> > > > <arg>-s</arg> > > > <arg>${wf:appPath()}/load/hql-delta-transform-${wf:id()}. > > hql</arg> > > > <arg>-n</arg> > > > <arg>${TARGET_DB_NAME}</arg> > > > <arg>-c</arg> > > > <arg>${LOAD_DIFF_DB_LOCATION}</arg> > > > <file>${HIVE_SITE_XML}</file> > > > <capture-output/> > > > </java> > > > <ok to="execute_load"/> > > > <error to="report_failure"/> > > > </action> > > > > > > > > > <action name="execute_load"> > > > <sub-workflow> > > > <app-path>${wf:appPath()}/load/load_sub_workflow-${wf: > > > id()}.xml</app-path> > > > <propagate-configuration/> > > > <configuration> > > > <property> > > > <name>jobName</name> > > > <value>${jobName}-load</value> > > > </property> > > > <property> > > > <name>parentWorkflowAppPath</name> > > > <value>${wf:appPath()}</value> > > > </property> > > > <property> > > > <name>kafkaConfigFilePath</name> > > > <value>${wf:appPath()}/load/kafka.properties</value> > > > </property> > > > <property> > > > <name>hqlDeltaTransformationPath</name> > > > <value>load/hql-delta-transform-${wf:id()}.hql</value> > > > </property> > > > </configuration> > > > </sub-workflow> > > > <ok to="generate_view_script"/> > > > <error to="report_failing_sub_wf"/> > > > </action> > > > > > > > > > > > > Sorry if I'm blabbing > > > /Pelle > > > > > > On Thu, Oct 20, 2016 at 3:52 AM, goun na <gou...@gmail.com> wrote: > > > > > > > Per Ullberg, a snippet of pom.xml would help us. :) > > > > Thanks, > > > > > > > > 2016-10-20 3:36 GMT+09:00 Per Ullberg <per.ullb...@klarna.com>: > > > > > > > > > @goun na: we keep one coordinator per (zip|war|jar) > > > > > > > > > > @shiva: I'm happy to share, but it's hard to know what you're in > need > > > of. > > > > > Ask and I will try to answer :) > > > > > > > > > > /Pelle > > > > > > > > > > > > > > > On Wednesday, October 19, 2016, Shiva Ramagopal <tr.s...@gmail.com > > > > > > wrote: > > > > > > > > > > > Per, > > > > > > > > > > > > Your approach seems very interesting. Could you elaborate more on > > > your > > > > > > approach? > > > > > > > > > > > > Thanks, > > > > > > Shiva > > > > > > > > > > > > On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg < > > per.ullb...@klarna.com > > > > > > <javascript:;>> wrote: > > > > > > > > > > > > > We package our oozie jobs with maven and release artifacts to > > > nexus. > > > > We > > > > > > > keep the version number as part of the coordinator name. That > way > > > we > > > > > have > > > > > > > full traceability between code base and running coordinators. > > > > > > > > > > > > > > regards > > > > > > > /Pelle > > > > > > > > > > > > > > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna < > > > > bafna.i...@gmail.com > > > > > > <javascript:;>> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Oozie does not have version control for jobs. When you > submit a > > > > > > > > workflow/coordinator/bundle to oozie, it stores it into DB > uses > > > it > > > > > from > > > > > > > > there for further execution. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Abhishek > > > > > > > > > On Oct 19, 2016, at 1:16 PM, goun na <gou...@gmail.com > > > > > > <javascript:;>> wrote: > > > > > > > > > > > > > > > > > > Hi users, > > > > > > > > > > > > > > > > > > What is the best to manage Oozie jobs? Is there a built-in > > > > version > > > > > > > > control > > > > > > > > > feature? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Goun Na > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Per Ullberg* > > > > > > > Data Vault Tech Lead > > > > > > > Odin Uppsala > > > > > > > +46 701612693 <+46+701612693> > > > > > > > > > > > > > > Klarna AB (publ) > > > > > > > Sveavägen 46, 111 34 Stockholm > > > > > > > Tel: +46 8 120 120 00 <+46812012000> > > > > > > > Reg no: 556737-0431 > > > > > > > klarna.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Per Ullberg* > > > > > Data Vault Tech Lead > > > > > Odin Uppsala > > > > > +46 701612693 <+46+701612693> > > > > > > > > > > Klarna AB (publ) > > > > > Sveavägen 46, 111 34 Stockholm > > > > > Tel: +46 8 120 120 00 <+46812012000> > > > > > Reg no: 556737-0431 > > > > > klarna.com > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Per Ullberg* > > > Data Vault Tech Lead > > > Odin Uppsala > > > +46 701612693 <+46+701612693> > > > > > > Klarna AB (publ) > > > Sveavägen 46, 111 34 Stockholm > > > Tel: +46 8 120 120 00 <+46812012000> > > > Reg no: 556737-0431 > > > klarna.com > > > > > > > > > -- > Peter Cseh > Software Engineer > <http://www.cloudera.com> >
