maven-assembly-plugin is good to use separate packaging. Thanks. :) 2016-10-20 15:05 GMT+09:00 Per Ullberg <per.ullb...@klarna.com>:
> Here's a pom, but I don't think it will tell you much. > > > <?xml version="1.0" encoding="UTF-8"?> > <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi=" > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > <parent> > <artifactId>hbase-to-hdfs-parent</artifactId> > <groupId>com.klarna.datavault</groupId> > <version>4.1.1-SNAPSHOT</version> > </parent> > <modelVersion>4.0.0</modelVersion> > > <artifactId>hbase-to-hdfs</artifactId> > > <dependencies> > > ... > > <dependency> > <groupId>com.github.spullara.mustache.java</groupId> > <artifactId>compiler</artifactId> > <version>0.8.18</version> > </dependency> > > <dependency> > <groupId>joda-time</groupId> > <artifactId>joda-time</artifactId> > <version>2.8</version> > <scope>test</scope> > </dependency> > <dependency> > <groupId>com.klarna</groupId> > <artifactId>hiverunner</artifactId> > <version>2.6.0</version> > <scope>test</scope> > <exclusions> > <exclusion> > <artifactId>junit</artifactId> > <groupId>junit</groupId> > </exclusion> > <exclusion> > <groupId>org.apache.calcite</groupId> > <artifactId>calcite-avatica</artifactId> > </exclusion> > <exclusion> > <groupId>org.apache.calcite</groupId> > <artifactId>calcite-core</artifactId> > </exclusion> > </exclusions> > </dependency> > </dependencies> > > > <build> > <resources> > <resource> > <directory>src/main/resources</directory> > <filtering>true</filtering> > </resource> > <resource> > <directory>src/main/templates</directory> > <filtering>true</filtering> > </resource> > </resources> > > <plugins> > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-assembly-plugin</artifactId> > <version>2.4</version> > <configuration> > <descriptor>assembly-descriptor.xml</descriptor> > </configuration> > <executions> > <execution> > <phase>package</phase> > <goals> > <goal>single</goal> > </goals> > </execution> > </executions> > </plugin> > </plugins> > </build> > > </project> > > > We use mustache a lot to materialize coordinators, workflows and hql, both > buildtime and runtime. A nice thing we came up with is that some of our > workflows have sub-workflows that gets materialized by the main workflow at > runtime. That way, different sub workflows may be generated dependent on a > runtime configuration. As an example we have sub workflows that load data > from either to postgres or to kafka. These two ways of loading are > described in two separate sub-workflow templates and are materialized at > runtime dependent on if the user configured a postgres or a kafka load. > > This way we can build workflows that can be more reactive to the current > state of the cluster and configuration. > > Here's an oozie snippet: > > <action name="materialize_load_sub_workflow" cred="hive_credentials"> > <java> > <prepare> > <mkdir path="${wf:appPath()}/load/"/> > </prepare> > <main-class>com.klarna.datavault.load.materialize. > MaterializeLoadSubWorkflowMain</main-class> > <!-- Output location for the materialized workflow XML --> > <arg>-o</arg> > <arg>${wf:appPath()}/load/load_sub_workflow-${wf:id()}.xml</arg> > <!-- Output location for the materialized kafka.properties > file for a Kafka Load --> > <arg>-k</arg> > <arg>${wf:appPath()}/load/kafka.properties</arg> > <arg>-w</arg> > <arg>${workflowConfigPath}</arg> > <arg>-d</arg> > <arg>${TARGET_DB_LOCATION}</arg> > <arg>-s</arg> > <arg>${wf:appPath()}/load/hql-delta-transform-${wf:id()}.hql</arg> > <arg>-n</arg> > <arg>${TARGET_DB_NAME}</arg> > <arg>-c</arg> > <arg>${LOAD_DIFF_DB_LOCATION}</arg> > <file>${HIVE_SITE_XML}</file> > <capture-output/> > </java> > <ok to="execute_load"/> > <error to="report_failure"/> > </action> > > > <action name="execute_load"> > <sub-workflow> > <app-path>${wf:appPath()}/load/load_sub_workflow-${wf: > id()}.xml</app-path> > <propagate-configuration/> > <configuration> > <property> > <name>jobName</name> > <value>${jobName}-load</value> > </property> > <property> > <name>parentWorkflowAppPath</name> > <value>${wf:appPath()}</value> > </property> > <property> > <name>kafkaConfigFilePath</name> > <value>${wf:appPath()}/load/kafka.properties</value> > </property> > <property> > <name>hqlDeltaTransformationPath</name> > <value>load/hql-delta-transform-${wf:id()}.hql</value> > </property> > </configuration> > </sub-workflow> > <ok to="generate_view_script"/> > <error to="report_failing_sub_wf"/> > </action> > > > > Sorry if I'm blabbing > /Pelle > > On Thu, Oct 20, 2016 at 3:52 AM, goun na <gou...@gmail.com> wrote: > > > Per Ullberg, a snippet of pom.xml would help us. :) > > Thanks, > > > > 2016-10-20 3:36 GMT+09:00 Per Ullberg <per.ullb...@klarna.com>: > > > > > @goun na: we keep one coordinator per (zip|war|jar) > > > > > > @shiva: I'm happy to share, but it's hard to know what you're in need > of. > > > Ask and I will try to answer :) > > > > > > /Pelle > > > > > > > > > On Wednesday, October 19, 2016, Shiva Ramagopal <tr.s...@gmail.com> > > wrote: > > > > > > > Per, > > > > > > > > Your approach seems very interesting. Could you elaborate more on > your > > > > approach? > > > > > > > > Thanks, > > > > Shiva > > > > > > > > On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <per.ullb...@klarna.com > > > > <javascript:;>> wrote: > > > > > > > > > We package our oozie jobs with maven and release artifacts to > nexus. > > We > > > > > keep the version number as part of the coordinator name. That way > we > > > have > > > > > full traceability between code base and running coordinators. > > > > > > > > > > regards > > > > > /Pelle > > > > > > > > > > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna < > > bafna.i...@gmail.com > > > > <javascript:;>> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Oozie does not have version control for jobs. When you submit a > > > > > > workflow/coordinator/bundle to oozie, it stores it into DB uses > it > > > from > > > > > > there for further execution. > > > > > > > > > > > > Thanks, > > > > > > Abhishek > > > > > > > On Oct 19, 2016, at 1:16 PM, goun na <gou...@gmail.com > > > > <javascript:;>> wrote: > > > > > > > > > > > > > > Hi users, > > > > > > > > > > > > > > What is the best to manage Oozie jobs? Is there a built-in > > version > > > > > > control > > > > > > > feature? > > > > > > > > > > > > > > Best, > > > > > > > Goun Na > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Per Ullberg* > > > > > Data Vault Tech Lead > > > > > Odin Uppsala > > > > > +46 701612693 <+46+701612693> > > > > > > > > > > Klarna AB (publ) > > > > > Sveavägen 46, 111 34 Stockholm > > > > > Tel: +46 8 120 120 00 <+46812012000> > > > > > Reg no: 556737-0431 > > > > > klarna.com > > > > > > > > > > > > > > > > > > -- > > > > > > *Per Ullberg* > > > Data Vault Tech Lead > > > Odin Uppsala > > > +46 701612693 <+46+701612693> > > > > > > Klarna AB (publ) > > > Sveavägen 46, 111 34 Stockholm > > > Tel: +46 8 120 120 00 <+46812012000> > > > Reg no: 556737-0431 > > > klarna.com > > > > > > > > > -- > > *Per Ullberg* > Data Vault Tech Lead > Odin Uppsala > +46 701612693 <+46+701612693> > > Klarna AB (publ) > Sveavägen 46, 111 34 Stockholm > Tel: +46 8 120 120 00 <+46812012000> > Reg no: 556737-0431 > klarna.com >
