http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/docs/src/site/twiki/DG_CustomActionExecutor.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_CustomActionExecutor.twiki b/docs/src/site/twiki/DG_CustomActionExecutor.twiki index 4acbf0d..7831484 100644 --- a/docs/src/site/twiki/DG_CustomActionExecutor.twiki +++ b/docs/src/site/twiki/DG_CustomActionExecutor.twiki @@ -66,6 +66,9 @@ Any configuration properties to be made available to this class should also be a The XML schema (XSD) for the new Actions should be added to oozie-site.xml, under the property 'oozie.service.WorkflowSchemaService.ext.schemas'. A comma separated list for multiple Action schemas. +The XML schema (XSD) for the new action should be also added to Fluent Job API. Please refer to +[[DG_FluentJobAPI#AE.C_Appendix_C_How_To_Extend][Fluent Job API :: How To Extend]] for details. + The executor class should be placed along with the oozie webapp in the correct path. Once Oozie is restarted, the custom action node can be used in workflows.
http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/docs/src/site/twiki/DG_Examples.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_Examples.twiki b/docs/src/site/twiki/DG_Examples.twiki index 13dfa28..5323a17 100644 --- a/docs/src/site/twiki/DG_Examples.twiki +++ b/docs/src/site/twiki/DG_Examples.twiki @@ -186,6 +186,13 @@ import java.util.Properties; Also asynchronous actions like FS action can be used / tested using =LocalOozie= / =OozieClient= API. Please see the module =oozie-mini= for details like =fs-decision.xml= workflow example. + +---++ Fluent Job API Examples + +There are some elaborate examples how to use the [[DG_FluentJobAPI][Fluent Job API]], under =examples/fluentjob/=. There are two +simple examples covered under [[DG_FluentJobAPI#A_Simple_Example][Fluent Job API :: A Simple Example]] and +[[DG_FluentJobAPI#A_More_Verbose_Example][Fluent Job API :: A More Verbose Example]]. + [[index][::Go back to Oozie Documentation Index::]] </noautolink> http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/docs/src/site/twiki/DG_FluentJobAPI.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_FluentJobAPI.twiki b/docs/src/site/twiki/DG_FluentJobAPI.twiki new file mode 100644 index 0000000..c8b764b --- /dev/null +++ b/docs/src/site/twiki/DG_FluentJobAPI.twiki @@ -0,0 +1,376 @@ +<noautolink> + +[[index][::Go back to Oozie Documentation Index::]] + +---+!! Fluent Job API + +%TOC% + +---++ Introduction + +Oozie is a mature workflow scheduler system. XML is the standard way of defining workflow, coordinator, or bundle jobs. For users +who prefer an alternative, the Fluent Job API provides a Java interface instead. + +---+++ Motivation + +Prior to Oozie 5.1.0, the following ways were available to submit a workflow, coordinator, or bundle job: through Oozie CLI or via +HTTP submit a generic workflow, coordinator, or bundle job, or submit a Pig, Hive, Sqoop, or MapReduce workflow job. + +As the generic way goes, the user has to have uploaded a workflow, coordinator, or bundle XML and all necessary dependencies like +scripts, JAR or ZIP files, to HDFS beforehand, as well as have a =job.properties= file at command line and / or provide any +missing parameters as part of the command. + +As the specific Pig, Hive, or Sqoop ways go, the user can provide all necessary parameters as part of the command issued. A + =workflow.xml= file will be generated with all the necessary details and stored to HDFS so that Oozie can grab it. Note that +dependencies have to be uploaded to HDFS beforehand as well. + +There are some usability problems by using the XML job definition. XML is not an ideal way to express dependencies and a directed +acyclic graph (DAG). We have to define a control flow, that is, which action follows the actual one. It's also necessary to build +the whole control flow up front as XML is a declarative language that doesn't allow for dynamic evaluation. We have to define also +boilerplate actions like start and end - those are present in every Oozie workflow, still need to explicitly define these. + +Apart from boilerplate actions, all the transitions between actions have also to be defined and taken care of. Furthermore, multiple +similar actions cannot inherit common properties from each other. Again, the reason being workflows are defined in XML. + +Fork and join actions have to be defined in pairs, that is, there shouldn't be defined a join those incoming actions do not share +the same ancestor fork. Such situations would result still in a DAG, but Oozie doesn't currently allow that. Note that with Fluent +Job API new dependencies are introduced automatically when the DAG represented by API code couldn't have been expressed as +fork / join pairs automatically. + +Either way, there were no programmatic ways to define workflow jobs. That doesn't mean users could not generate XML themselves - +actually this is something HUE's Oozie UI also tries to target. + +---+++ Goals + +Fluent Job API aims to solve following from the user's perspective. It provides a Java API instead of declarative XML to define +workflows. It defines dependencies across actions as opposed to defining a control flow. This is how data engineers and data +scientists think. It eliminates all boilerplate actions and transitions. Only the necessary bits should be defined. + +Multiple similar actions can inherit from each other. In fact, since Fluent Job API is programmatic, it's possible to generate +actions or even workflows using conditional, iterative, or recursive structures. + +Fluent Job API is backwards compatible with workflows defined as XML. That is, it should also be possible to have a Fluent Job API +workflow rendered as XML, as well as coexist XML based and Fluent Job API based workflows in the same Oozie installation at the same +time all workflow action types. When XSDs change, as few manual steps are necessary as possible both on API internal and public +side. + +---+++ Non-goals + +The following points are not targeted for the initial release of Fluent Job API with Oozie 5.1.0. It doesn't provide API in any +language other than Java. It doesn't provide a REPL. It doesn't allow for dynamic action instantiation depending on e.g. conditional +logic. That is, using the API users still have to implement the whole workflow generation logic in advance. + +It has no support for programmatic coordinators and bundles, or even EL expressions created by API builders. Note that EL +expressions for workflows can now be expressed the way these are used in XML workflow definitions, as strings in the right places. + +At the moment only the transformation from Fluent Job API to workflow definition is present. The other direction, from workflow +definition to Fluent Job API JAR artifact, though sensible, is not supported. + +It's based only on latest XSDs. Older XSD versions, as well as conversion between XSD versions are not supported. Also no support +for user-supplied custom actions / XSDs. + +Most of the non-goals may be targeted as enhancements of the Fluent Job API for future Oozie releases. + +---+++ Approach + +When using the Fluent Job API, the following points are different from the XML jobs definition. Instead of control flow (successor) +definition, the user can define dependencies (parents of an action). + +All boilerplate (start, end, ...) has been eliminated, only nodes having useful actions have to be defined. + +Control flow and necessary boilerplate are generated automatically by keeping user defined dependencies, and possibly introducing +new dependencies to keep Oozie workflow format of nested fork / join pairs. Note that not every dependency DAG can be expressed in +the Oozie workflow format. When this is not possible, user is notified at build time. + +---++ How To Use + +---+++ A Simple Example + +The simplest thing to create using the Oozie Fluent Job API is a workflow consisting of only one action. Let's see how it goes, step +by step. + +First, put the project =org.apache.oozie:oozie-fluent-job-api= to the build path. In case of a Maven managed build, create a new +Maven project and declare a Maven dependency to =org.apache.oozie:oozie-fluent-job-api=. + +Then, create a class that =implements WorkflowFactory= and implement the method =WorkflowFactory#create()=. inside that method, +create a =ShellAction= using =ShellActionBuilder=, fill in some attributes then create a =Workflow= using =WorkflowBuilder= using +the =ShellAction= just built. Return the =Workflow=. + +Compile a Fluent Job API jar that has the =Main-Class= attribute set to the =WorkflowFactory= subclass just created, +e.g. =shell-workflow.jar=. + +Moving on, [[DG_CommandLineTool#Checking_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file][check via command line]] that +the compiled API JAR file is valid. + +As a finishing touch, +[[DG_CommandLineTool#Running_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file][run via command line]] the Fluent Job API +workflow. + +*For reference, a simplistic API JAR example consisting of a =Workflow= having only one =ShellAction=:* +<verbatim> +public class MyFirstWorkflowFactory implements WorkflowFactory { +. + @Override + public Workflow create() { + final ShellAction shellAction = ShellActionBuilder.create() + .withName("shell-action") + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withConfigProperty("mapred.job.queue.name", "${queueName}") + .withExecutable("echo") + .withArgument("my_output=Hello Oozie") + .withCaptureOutput(true) + .build(); +. + final Workflow shellWorkflow = new WorkflowBuilder() + .withName("shell-workflow") + .withDagContainingNode(shellAction).build(); +. + return shellWorkflow; + } +} +</verbatim> + +*After check, the generated workflow XML looks like this:* +<verbatim> +<?xml version="1.0" encoding="UTF-8" standalone="yes"?> +<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" xmlns:shell="uri:oozie:shell-action:1.0" name="shell-workflow"> +. + <workflow:start to="parent"/> +. + <workflow:kill name="kill"> + <workflow:message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</workflow:message> + </workflow:kill> +. + <workflow:action name="shell-action"> + <shell:shell> + <shell:resource-manager>${resourceManager}</shell:resource-manager> + <shell:name-node>${nameNode}</shell:name-node> + <shell:configuration> + <shell:property> + <shell:name>mapred.job.queue.name</shell:name> + <shell:value>${queueName}</shell:value> + </shell:property> + </shell:configuration> + <shell:exec>echo</shell:exec> + <shell:argument>my_output=Hello Oozie</shell:argument> + <shell:capture-output/> + </shell:shell> + <workflow:ok to="end"/> + <workflow:error to="kill"/> + </workflow:action> +. + <workflow:end name="end"/> +. +</workflow:workflow-app> +</verbatim> + + +---+++ A More Verbose Example + +*Error handling* + +If you would like to provide some error handling in case of action failure, you should add an =ErrorHandler= to the =Node= +representing the action. The error handler action will be added as the ="error-transition"= of the original action in the generated +Oozie workflow XML. Both the ="ok-transition"= and the ="error-transition"= of the error handler action itself will lead to an +autogenerated kill node. + +*Here you find an example consisting of a =Workflow= having three =ShellAction=s, an error handler =EmailAction=, and one =decision= +to sort out which way to go:* +<verbatim> +public class MySecondWorkflowFactory implements WorkflowFactory { +. + @Override + public Workflow create() { + final ShellAction parent = ShellActionBuilder.create() + .withName("parent") + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withConfigProperty("mapred.job.queue.name", "${queueName}") + .withExecutable("echo") + .withArgument("my_output=Hello Oozie") + .withCaptureOutput(true) + .withErrorHandler(ErrorHandler.buildAsErrorHandler(EmailActionBuilder.create() + .withName("email-on-error") + .withRecipient("[email protected]") + .withSubject("Workflow error") + .withBody("Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]"))) + .build(); +. + ShellActionBuilder.createFromExistingAction(parent) + .withName("happy-path") + .withParentWithCondition(parent, "${wf:actionData('parent')['my_output'] eq 'Hello Oozie'}") + .withoutArgument("my_output=Hello Oozie") + .withArgument("Happy path") + .withCaptureOutput(null) + .build(); +. + ShellActionBuilder.createFromExistingAction(parent) + .withName("sad-path") + .withParentDefaultConditional(parent) + .withArgument("Sad path") + .build(); +. + final Workflow workflow = new WorkflowBuilder() + .withName("shell-example") + .withDagContainingNode(parent).build(); +. + return workflow; + } +} +</verbatim> + +*After check, the generated workflow XML looks like this:* +<verbatim> +<?xml version="1.0" encoding="UTF-8" standalone="yes"?> +<workflow:workflow-app ... name="shell-example"> +. + <workflow:start to="parent"/> +. + <workflow:kill name="kill"> + <workflow:message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</workflow:message> + </workflow:kill> +. + <workflow:action name="email-on-error"> + <email:email> + <email:to>[email protected]</email:to> + <email:subject>Workflow error</email:subject> + <email:body>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</email:body> + </email:email> + <workflow:ok to="kill"/> + <workflow:error to="kill"/> + </workflow:action> +. + <workflow:action name="parent"> + <shell:shell> + <shell:resource-manager>${resourceManager}</shell:resource-manager> + <shell:name-node>${nameNode}</shell:name-node> + <shell:configuration> + <shell:property> + <shell:name>mapred.job.queue.name</shell:name> + <shell:value>${queueName}</shell:value> + </shell:property> + </shell:configuration> + <shell:exec>echo</shell:exec> + <shell:argument>my_output=Hello Oozie</shell:argument> + <shell:capture-output/> + </shell:shell> + <workflow:ok to="decision1"/> + <workflow:error to="email-on-error"/> + </workflow:action> +. + <workflow:decision name="decision1"> + <workflow:switch> + <workflow:case to="happy-path">${wf:actionData('parent')['my_output'] eq 'Hello Oozie'}</workflow:case> + <workflow:default to="sad-path"/> + </workflow:switch> + </workflow:decision> +. + <workflow:action name="happy-path"> + <shell:shell> + <shell:resource-manager>${resourceManager}</shell:resource-manager> + <shell:name-node>${nameNode}</shell:name-node> + <shell:configuration> + <shell:property> + <shell:name>mapred.job.queue.name</shell:name> + <shell:value>${queueName}</shell:value> + </shell:property> + </shell:configuration> + <shell:exec>echo</shell:exec> + <shell:argument>Happy path</shell:argument> + </shell:shell> + <workflow:ok to="end"/> + <workflow:error to="email-on-error"/> + </workflow:action> +. + <workflow:action name="sad-path"> + <shell:shell> + <shell:resource-manager>${resourceManager}</shell:resource-manager> + <shell:name-node>${nameNode}</shell:name-node> + <shell:configuration> + <shell:property> + <shell:name>mapred.job.queue.name</shell:name> + <shell:value>${queueName}</shell:value> + </shell:property> + </shell:configuration> + <shell:exec>echo</shell:exec> + <shell:argument>my_output=Hello Oozie</shell:argument> + <shell:argument>Sad path</shell:argument> + <shell:capture-output/> + </shell:shell> + <workflow:ok to="end"/> + <workflow:error to="email-on-error"/> + </workflow:action> +. + <workflow:end name="end"/> +. +</workflow:workflow-app> +</verbatim> + +---+++ Runtime Limitations + +Even if Fluent Job API tries to abstract away the task of assembly job descriptor XML files, there are some runtime +limitations apart from the [[DG_FluentJobAPI#Non-goals][non-goals section]]. All such limitations are based on the current +implementations and subject to further improvements and fixes. + +There is only one =kill= possibility in every =workflow=. That is, there can be defined only one =action= to be executed just before +any other =action= turns to be =kill=ed. Furthermore, =kill= goes to =end= directly. That means, there cannot be defined an +intricate network of =kill= nodes, cascading sometimes to other =action= nodes, avoiding going to =end= in the first place. + +There are places where =decision= node generation fails, throwing an =Exception=. The problem is that during the transformation, +Fluent Job API reaches a state where there is a =fork= that transitions to two =decision= nodes, which in turn split into two paths +each. One of the paths from the first =decision= joins a path from the other =decision=, but the remaining conditional paths never +meet. Therefore, not all paths originating from the =fork= converge to the same =join=. + +---++ Appendixes + +---+++ AE.A Appendix A, API JAR format + +It's kept simple - all the necessary Java class files that are needed are packed into a JAR file, that has a =META-INF/MANIFEST.MF= +with a single entry having the =Main-Class= attribute set to the fully qualified name of the entry class, the one that +=implements WorkflowFactory=: +<verbatim> +Main-Class: org.apache.oozie.jobs.api.factory.MyFirstWorkflowFactory +</verbatim> + +*An example of the command line assembly of such an API JAR:* +<verbatim> +jar cfe simple-workflow.jar org.apache.oozie.fluentjob.api.factory.MyFirstWorkflowFactory \ +-C /Users/forsage/Workspace/oozie/fluent-job/fluent-job-api/target/classes \ +org/apache/oozie/jobs/api/factory/MyFirstWorkflowFactory.class +</verbatim> + +---+++ AE.B Appendix B, Some Useful Builder classes + +For a complete list of =Builder= classes, please have a look at =oozie-fluent-job-api= artifact's following packages: + * =org.apache.oozie.fluentjob.api.action= - =ActionBuilder= classes + * =org.apache.oozie.fluentjob.api.factory= - the single entry point, =WorkflowFactory= is here + * =org.apache.oozie.fluentjob.api.workflow= - workflow related =Builder= classes + +On examples how to use these please see =oozie-examples= artifact's =org.apache.oozie.example.fluentjob= package. + +---+++ AE.C Appendix C, How To Extend + +Sometimes there are new XSD versions of an existing custom or core workflow action, sometimes it's a new custom workflow action that +gets introduced. In any case, Fluent Job API needs to keep up with the changes. + +Here are the steps needed: + * in =fluent-job-api/pom.xml= extend or modify =jaxb2-maven-plugin= section =sources= by a new =source= + * in =fluent-job-api/src/main/xjb/bindings.xml= extend by a new or modify an existing =jaxb:bindings= + * in =fluent-job-api=, =org.apache.oozie.fluentjob.api.mapping= package, introduce a new or modify an existing =DozerConverter= + * in =dozer_config.xml=, introduce a new or modify an existing =converter= inside =custom-converters= + * in =fluent-job-api=, =org.apache.oozie.fluentjob.api.action=, introduce a new =Action= and a new =Builder= + * write new / modify existing relevant unit and integration tests + +---+++ AE.D Appendix D, API compatibility guarantees + +Fluent Job API is available beginning version 5.1.0. It's marked [email protected]= (intended for use in Oozie itself) and [email protected]= (no stability guarantees are provided across any level of release granularity) to indicate that for +the next few minor releases it's bound to change a lot. + +Beginning from around 5.4.0 planning the next phase, [email protected]= (compatibility breaking only between minors), +and a few minor releases later, [email protected]= (safe to use outside of Oozie). + +[[index][::Go back to Oozie Documentation Index::]] + +</noautolink> http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/pom.xml ---------------------------------------------------------------------- diff --git a/examples/pom.xml b/examples/pom.xml index 420b4f8..680e3fb 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -87,6 +87,12 @@ </dependency> <dependency> + <groupId>org.apache.oozie</groupId> + <artifactId>oozie-fluent-job-api</artifactId> + <scope>compile</scope> + </dependency> + + <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <scope>test</scope> http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java ---------------------------------------------------------------------- diff --git a/examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java b/examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java new file mode 100644 index 0000000..b450e75 --- /dev/null +++ b/examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java @@ -0,0 +1,93 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.example.fluentjob; + +import com.google.common.collect.Lists; +import org.apache.oozie.fluentjob.api.action.*; +import org.apache.oozie.fluentjob.api.factory.WorkflowFactory; +import org.apache.oozie.fluentjob.api.workflow.*; + +/** + * This {@link WorkflowFactory} generates a workflow definition that has {@code credentials} section, and an action that has + * {@code retry-interval}, {@code retry-max}, and {@code retry-policy} attributes set. + * <p> + * Note that {@link WorkflowBuilder#withCredentials(Credentials)} doesn't necessarily have to be called, since if + * {@link WorkflowBuilder#credentialsBuilder} is emtpy by the time {@link WorkflowBuilder#build()} is called, + * {@link Workflow#credentials} is built based on all the {@link Node#getCredentials()} that have been added to that + * {@code Workflow} in beforehand. + * <p> + * Note also that when {@code WorkflowBuilder#withCredentials(Credentials)} is explicitly called, the {@code <workflowapp />}'s + * {@code <credential />} section is generated only by using the {@code Credentials} defined on the {@code Workflow} level. + * <p> + * This way, users can, if they want to, omit calling {@code WorkflowBuilder#withCredentials(Credentials)} by default, but can + * also override the autogenerated {@code <credentials />} section of {@code <workflowapp />} by explicitly calling that method + * after another call to {@link CredentialsBuilder#build()}. + * {@link CredentialsBuilder#build()}. + */ +public class CredentialsRetrying implements WorkflowFactory { + @Override + public Workflow create() { + final Credential hbaseCredential = CredentialBuilder.create() + .withName("hbase") + .withType("hbase") + .build(); + + final Credential hcatalogCredential = CredentialBuilder.create() + .withName("hcatalog") + .withType("hcatalog") + .withConfigurationEntry("hcat.metastore.uri", "thrift://<host>:<port>") + .withConfigurationEntry("hcat.metastore.principal", "hive/<host>@<realm>") + .build(); + + final Credential hive2Credential = CredentialBuilder.create() + .withName("hive2") + .withType("hive2") + .withConfigurationEntry("jdbcUrl", "jdbc://<host>/<database>") + .build(); + + final ShellAction shellActionWithHBase = ShellActionBuilder + .create() + .withName("shell-with-hbase-credential") + .withCredential(hbaseCredential) + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withExecutable("call-hbase.sh") + .build(); + + Hive2ActionBuilder + .createFromExistingAction(shellActionWithHBase) + .withParent(shellActionWithHBase) + .withName("hive2-action-with-hcatalog-and-hive2-credentials") + .clearCredentials() + .withCredential(hcatalogCredential) + .withCredential(hive2Credential) + .withRetryInterval(1) + .withRetryMax(3) + .withRetryPolicy("exponential") + .withScript("call-hive2.sql") + .build(); + + final Workflow workflow = new WorkflowBuilder() + .withName("workflow-with-credentials") + .withDagContainingNode(shellActionWithHBase) + .build(); + + return workflow; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java ---------------------------------------------------------------------- diff --git a/examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java b/examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java new file mode 100644 index 0000000..c50fda0 --- /dev/null +++ b/examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java @@ -0,0 +1,49 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.example.fluentjob; + +import org.apache.oozie.fluentjob.api.action.LauncherBuilder; +import org.apache.oozie.fluentjob.api.factory.WorkflowFactory; +import org.apache.oozie.fluentjob.api.workflow.GlobalBuilder; +import org.apache.oozie.fluentjob.api.workflow.Workflow; +import org.apache.oozie.fluentjob.api.workflow.WorkflowBuilder; + +/** + * This {@link WorkflowFactory} generates a workflow definition that has global section. + */ +public class Global implements WorkflowFactory { + @Override + public Workflow create() { + final Workflow workflow = new WorkflowBuilder() + .withName("workflow-with-global") + .withGlobal(GlobalBuilder.create() + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withJobXml("job.xml") + .withConfigProperty("key1", "value1") + .withLauncher(new LauncherBuilder() + .withMemoryMb(1024L) + .withVCores(1L) + .build()) + .build()) + .build(); + + return workflow; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java ---------------------------------------------------------------------- diff --git a/examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java b/examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java new file mode 100644 index 0000000..a8948fd --- /dev/null +++ b/examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java @@ -0,0 +1,56 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.example.fluentjob; + +import org.apache.oozie.fluentjob.api.action.EmailActionBuilder; +import org.apache.oozie.fluentjob.api.action.ErrorHandler; +import org.apache.oozie.fluentjob.api.action.JavaAction; +import org.apache.oozie.fluentjob.api.action.JavaActionBuilder; +import org.apache.oozie.fluentjob.api.factory.WorkflowFactory; +import org.apache.oozie.fluentjob.api.workflow.Workflow; +import org.apache.oozie.fluentjob.api.workflow.WorkflowBuilder; + +/** + * This {@link WorkflowFactory} generates the exact same workflow definition as {@code apps/java-main/workflow.xml}. + */ +public class JavaMain implements WorkflowFactory { + @Override + public Workflow create() { + final JavaAction parent = JavaActionBuilder.create() + .withName("java-main") + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withConfigProperty("mapred.job.queue.name", "${queueName}") + .withMainClass("org.apache.oozie.example.DemoJavaMain") + .withArg("Hello") + .withArg("Oozie!") + .withErrorHandler(ErrorHandler.buildAsErrorHandler(EmailActionBuilder.create() + .withName("email-on-error") + .withRecipient("[email protected]") + .withSubject("Workflow error") + .withBody("Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]"))) + .build(); + + final Workflow workflow = new WorkflowBuilder() + .withName("java-main-example") + .withDagContainingNode(parent).build(); + + return workflow; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java ---------------------------------------------------------------------- diff --git a/examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java b/examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java new file mode 100644 index 0000000..e1185b1 --- /dev/null +++ b/examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java @@ -0,0 +1,79 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.example.fluentjob; + +import org.apache.oozie.fluentjob.api.action.EmailActionBuilder; +import org.apache.oozie.fluentjob.api.action.ErrorHandler; +import org.apache.oozie.fluentjob.api.factory.WorkflowFactory; +import org.apache.oozie.fluentjob.api.workflow.Workflow; +import org.apache.oozie.fluentjob.api.workflow.WorkflowBuilder; +import org.apache.oozie.fluentjob.api.action.ShellAction; +import org.apache.oozie.fluentjob.api.action.ShellActionBuilder; + +/** + * An easily understandable {@link WorkflowFactory} that creates a {@link Workflow} instance consisting of + * multiple {@link ShellAction}s, the latter depending conditionally on the output of the former. + * <p> + * It demonstrates how the Jobs API can be used to create dynamic {@code Workflow} artifacts, as well as + * serves as an input for {@code TestOozieCLI} methods that check, submit or run Jobs API {@code .jar} files. + */ +public class MultipleShellActions implements WorkflowFactory { + + @Override + public Workflow create() { + final ShellAction parent = ShellActionBuilder.create() + .withName("parent") + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withConfigProperty("mapred.job.queue.name", "${queueName}") + .withArgument("my_output=Hello Oozie") + .withExecutable("echo") + .withCaptureOutput(true) + .withErrorHandler(ErrorHandler.buildAsErrorHandler(EmailActionBuilder.create() + .withName("email-on-error") + .withRecipient("[email protected]") + .withSubject("Workflow error") + .withBody("Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]"))) + .build(); + + ShellAction next = parent; + + for (int ixShellPair = 0; ixShellPair < 5; ixShellPair++) { + final ShellAction happyPath = ShellActionBuilder.createFromExistingAction(parent) + .withName("happy-path-" + ixShellPair) + .withParentWithCondition(next, "${wf:actionData('" + next.getName() + "')['my_output'] eq 'Hello Oozie'}") + .build(); + + ShellActionBuilder.createFromExistingAction(parent) + .withName("sad-path-" + ixShellPair) + .withParentDefaultConditional(next) + .withArgument("Sad path " + ixShellPair) + .withCaptureOutput(null) + .build(); + + next = happyPath; + } + + final Workflow workflow = new WorkflowBuilder() + .withName("shell-example") + .withDagContainingNode(parent).build(); + + return workflow; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java ---------------------------------------------------------------------- diff --git a/examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java b/examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java new file mode 100644 index 0000000..18311be --- /dev/null +++ b/examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java @@ -0,0 +1,39 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.example.fluentjob; + +import org.apache.oozie.fluentjob.api.factory.WorkflowFactory; +import org.apache.oozie.fluentjob.api.workflow.Workflow; +import org.apache.oozie.fluentjob.api.workflow.WorkflowBuilder; + +/** + * This {@link WorkflowFactory} generates a workflow definition that has parameters section. + */ +public class Parameters implements WorkflowFactory { + @Override + public Workflow create() { + final Workflow workflow = new WorkflowBuilder() + .withName("workflow-with-parameters") + .withParameter("name1", "value1") + .withParameter("name2", "value2", "description2") + .build(); + + return workflow; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java ---------------------------------------------------------------------- diff --git a/examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java b/examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java new file mode 100644 index 0000000..b20a92a --- /dev/null +++ b/examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.example.fluentjob; + +import org.apache.oozie.fluentjob.api.action.*; +import org.apache.oozie.fluentjob.api.factory.WorkflowFactory; +import org.apache.oozie.fluentjob.api.workflow.Workflow; +import org.apache.oozie.fluentjob.api.workflow.WorkflowBuilder; + +/** + * This {@link WorkflowFactory} generates a similar workflow definition to {@code apps/shell/workflow.xml}. + */ +public class Shell implements WorkflowFactory { + @Override + public Workflow create() { + final ShellAction parent = ShellActionBuilder.create() + .withName("parent") + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withConfigProperty("mapred.job.queue.name", "${queueName}") + .withArgument("my_output=Hello Oozie") + .withExecutable("echo") + .withCaptureOutput(true) + .withErrorHandler(ErrorHandler.buildAsErrorHandler(EmailActionBuilder.create() + .withName("email-on-error") + .withRecipient("[email protected]") + .withSubject("Workflow error") + .withBody("Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]"))) + .build(); + + ShellActionBuilder.createFromExistingAction(parent) + .withName("happy-path") + .withParentWithCondition(parent, "${wf:actionData('parent')['my_output'] eq 'Hello Oozie'}") + .withoutArgument("my_output=Hello Oozie") + .withArgument("Happy path") + .withCaptureOutput(null) + .build(); + + ShellActionBuilder.createFromExistingAction(parent) + .withName("sad-path") + .withParentDefaultConditional(parent) + .withArgument("Sad path") + .build(); + + final Workflow workflow = new WorkflowBuilder() + .withName("shell-example") + .withDagContainingNode(parent).build(); + + return workflow; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java ---------------------------------------------------------------------- diff --git a/examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java b/examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java new file mode 100644 index 0000000..6065933 --- /dev/null +++ b/examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java @@ -0,0 +1,61 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.example.fluentjob; + +import org.apache.oozie.fluentjob.api.action.*; +import org.apache.oozie.fluentjob.api.factory.WorkflowFactory; +import org.apache.oozie.fluentjob.api.workflow.Workflow; +import org.apache.oozie.fluentjob.api.workflow.WorkflowBuilder; + +/** + * This {@link WorkflowFactory} generates the exact same workflow definition as {@code apps/spark/workflow.xml}. + */ +public class Spark implements WorkflowFactory { + @Override + public Workflow create() { + final Prepare prepare = new PrepareBuilder() + .withDelete("${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/spark") + .build(); + + final SparkAction parent = SparkActionBuilder.create() + .withName("spark-file-copy") + .withResourceManager("${resourceManager}") + .withNameNode("${nameNode}") + .withPrepare(prepare) + .withConfigProperty("mapred.job.queue.name", "${queueName}") + .withArg("${nameNode}/user/${wf:user()}/${examplesRoot}/input-data/text/data.txt") + .withArg("${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/spark") + .withMaster("${master}") + .withActionName("Spark File Copy Example") + .withActionClass("org.apache.oozie.example.SparkFileCopy") + .withJar("${nameNode}/user/${wf:user()}/${examplesRoot}/apps/spark/lib/oozie-examples.jar") + .withErrorHandler(ErrorHandler.buildAsErrorHandler(EmailActionBuilder.create() + .withName("email-on-error") + .withRecipient("[email protected]") + .withSubject("Workflow error") + .withBody("Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]"))) + .build(); + + final Workflow workflow = new WorkflowBuilder() + .withName("spark-file-copy") + .withDagContainingNode(parent).build(); + + return workflow; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/findbugs-filter.xml ---------------------------------------------------------------------- diff --git a/findbugs-filter.xml b/findbugs-filter.xml index 03ee4d1..133178f 100644 --- a/findbugs-filter.xml +++ b/findbugs-filter.xml @@ -29,4 +29,16 @@ <Method name="toString" /> <Bug pattern="WMI_WRONG_MAP_ITERATOR" /> </Match> + + <!-- Don't have much control on generated JAXB2 classes --> + <Match> + <Package name="~org\.apache\.oozie\.fluentjob\.api\.generated.*" /> + <Bug pattern="EQ_UNUSUAL" /> + </Match> + + <!-- Directory name is private static final --> + <Match> + <Class name="org.apache.oozie.fluentjob.api.GraphVisualization" /> + <Bug pattern="WEAK_FILENAMEUTILS" /> + </Match> </FindBugsFilter> http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/fluent-job/fluent-job-api/pom.xml ---------------------------------------------------------------------- diff --git a/fluent-job/fluent-job-api/pom.xml b/fluent-job/fluent-job-api/pom.xml new file mode 100644 index 0000000..4c9b853 --- /dev/null +++ b/fluent-job/fluent-job-api/pom.xml @@ -0,0 +1,212 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> + <modelVersion>4.0.0</modelVersion> + + <parent> + <groupId>org.apache.oozie</groupId> + <artifactId>oozie-fluent-job</artifactId> + <version>5.1.0-SNAPSHOT</version> + </parent> + + <artifactId>oozie-fluent-job-api</artifactId> + <version>5.1.0-SNAPSHOT</version> + <description>Apache Oozie Fluent Job API</description> + <name>Apache Oozie Fluent Job API</name> + <packaging>jar</packaging> + + <dependencies> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.mockito</groupId> + <artifactId>mockito-all</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.commons</groupId> + <artifactId>commons-lang3</artifactId> + </dependency> + <dependency> + <groupId>guru.nidi</groupId> + <artifactId>graphviz-java</artifactId> + </dependency> + <dependency> + <groupId>net.sf.dozer</groupId> + <artifactId>dozer</artifactId> + </dependency> + <dependency> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + <scope>compile</scope> + </dependency> + <dependency> + <groupId>org.jvnet.jaxb2_commons</groupId> + <artifactId>jaxb2-basics</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-annotations</artifactId> + </dependency> + </dependencies> + + <build> + <plugins> + <plugin> + <groupId>org.apache.rat</groupId> + <artifactId>apache-rat-plugin</artifactId> + <configuration> + <excludeSubProjects>false</excludeSubProjects> + <excludes> + <!-- excluding all as the root POM does the full check--> + <exclude>**</exclude> + </excludes> + </configuration> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-jar-plugin</artifactId> + <executions> + <execution> + <goals> + <goal>test-jar</goal> + </goals> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-assembly-plugin</artifactId> + <configuration> + <descriptors> + <descriptor>../../src/main/assemblies/tools.xml</descriptor> + </descriptors> + </configuration> + </plugin> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>jaxb2-maven-plugin</artifactId> + <executions> + <execution> + <id>xjc</id> + <goals> + <goal>xjc</goal> + </goals> + </execution> + </executions> + <configuration> + <arguments> + <argument>-XhashCode</argument> + <argument>-Xequals</argument> + <argument>-Xnamespace-prefix</argument> + <argument>-Xsetters</argument> + </arguments> + + <sources> + <source>../../client/src/main/resources/distcp-action-1.0.xsd</source> + <source>../../client/src/main/resources/email-action-0.2.xsd</source> + <source>../../client/src/main/resources/hive2-action-1.0.xsd</source> + <source>../../client/src/main/resources/hive-action-1.0.xsd</source> + <source>../../client/src/main/resources/oozie-sla-0.2.xsd</source> + <source>../../client/src/main/resources/oozie-workflow-1.0.xsd</source> + <source>../../client/src/main/resources/shell-action-1.0.xsd</source> + <source>../../client/src/main/resources/spark-action-1.0.xsd</source> + <source>../../client/src/main/resources/sqoop-action-1.0.xsd</source> + <source>../../client/src/main/resources/ssh-action-0.2.xsd</source> + </sources> + </configuration> + + <dependencies> + <dependency> + <groupId>org.jvnet.jaxb2_commons</groupId> + <artifactId>jaxb2-basics</artifactId> + <version>1.11.1</version> + </dependency> + <dependency> + <groupId>org.jvnet.jaxb2_commons</groupId> + <artifactId>jaxb2-namespace-prefix</artifactId> + <version>1.3</version> + </dependency> + </dependencies> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-javadoc-plugin</artifactId> + <configuration> + <excludePackageNames>org.apache.oozie.fluentjob.api.generated.*</excludePackageNames> + </configuration> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-checkstyle-plugin</artifactId> + <configuration> + <excludes>**/generated/**/*</excludes> + </configuration> + </plugin> + </plugins> + </build> + + <profiles> + <profile> + <id>generateDocs</id> + <activation> + <activeByDefault>false</activeByDefault> + <property> + <name>generateDocs</name> + </property> + </activation> + <build> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-javadoc-plugin</artifactId> + <configuration> + <linksource>true</linksource> + <quiet>true</quiet> + <verbose>false</verbose> + <source>${maven.compile.source}</source> + <charset>${maven.compile.encoding}</charset> + <groups> + <group> + <title>Fluent Job API</title> + <packages> + org.apache.oozie.fluentjob.api + </packages> + </group> + </groups> + <excludePackageNames>org.apache.oozie.fluentjob.api.generated.*</excludePackageNames> + </configuration> + <executions> + <execution> + <goals> + <goal>javadoc</goal> + </goals> + <phase>package</phase> + </execution> + </executions> + </plugin> + </plugins> + </build> + </profile> + </profiles> +</project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java ---------------------------------------------------------------------- diff --git a/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java new file mode 100644 index 0000000..dd18f7a --- /dev/null +++ b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java @@ -0,0 +1,102 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.fluentjob.api; + +import com.google.common.base.Preconditions; + +/** + * A class representing a condition in the "switch statement" of an Oozie decision node. + */ +public class Condition { + private final String condition; + private final boolean isDefault; + + private Condition(final String condition, final boolean isDefault) { + final boolean bothFieldsSet = condition == null && !isDefault; + final boolean bothFieldsUnset = condition != null && isDefault; + Preconditions.checkArgument(!bothFieldsSet && !bothFieldsUnset, + "Exactly one of 'condition' and 'isDefault' must be non-null or true (respectively)."); + + this.condition = condition; + this.isDefault = isDefault; + } + + /** + * Creates an actual condition (as opposed to a default condition). + * @param condition The string defining the condition. + * @return A new actual condition. + */ + public static Condition actualCondition(final String condition) { + Preconditions.checkArgument(condition != null, "The argument 'condition' must not be null."); + + return new Condition(condition, false); + } + + /** + * Creates a new default condition. Every decision node must have a default path which is chosen if no other + * condition is true. + * @return A new default condition. + */ + public static Condition defaultCondition() { + return new Condition(null, true); + } + + /** + * Returns the string defining the condition or {@code null} if this is a default condition. + * @return The string defining the condition or {@code null} if this is a default condition. + */ + public String getCondition() { + return condition; + } + + /** + * Returns whether this condition is a default condition. + * @return {@code true} if this condition is a default condition, {@code false} otherwise. + */ + public boolean isDefault() { + return isDefault; + } + + @Override + public boolean equals(final Object o) { + if (this == o) { + return true; + } + + if (o == null || getClass() != o.getClass()) { + return false; + } + + final Condition other = (Condition) o; + + if (isDefault != other.isDefault) { + return false; + } + + return condition != null ? condition.equals(other.condition) : other.condition == null; + } + + @Override + public int hashCode() { + int result = condition != null ? condition.hashCode() : 0; + result = 31 * result + (isDefault ? 1 : 0); + + return result; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java ---------------------------------------------------------------------- diff --git a/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java new file mode 100644 index 0000000..437d6b3 --- /dev/null +++ b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java @@ -0,0 +1,117 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.fluentjob.api; + +import guru.nidi.graphviz.engine.Format; +import guru.nidi.graphviz.engine.Graphviz; +import guru.nidi.graphviz.model.MutableGraph; +import guru.nidi.graphviz.parse.Parser; +import org.apache.commons.io.FilenameUtils; +import org.apache.oozie.fluentjob.api.action.Node; +import org.apache.oozie.fluentjob.api.dag.Decision; +import org.apache.oozie.fluentjob.api.dag.Graph; +import org.apache.oozie.fluentjob.api.dag.NodeBase; +import org.apache.oozie.fluentjob.api.workflow.Workflow; + +import java.io.File; +import java.io.IOException; +import java.util.Collection; +import java.util.List; + +/** + * Given a {@link Graph} or a {@link Workflow} instance, creates a visually appealing output + * using {@code nidi-graphviz} library in either {@code .dot} or {@code .png} format into {@link #PARENT_FOLDER_NAME}. + * <p> + * Applying memory constraints, the width of the resulting {@code .png} is limited to {@link #PNG_WIDTH}. + */ +public class GraphVisualization { + private static final String PARENT_FOLDER_NAME = "target/graphviz"; + private static final int PNG_WIDTH = 1024; + + public static String graphToDot(final Graph graph) { + return nodeBasesToDot(graph.getNodes()); + } + + private static String nodeBasesToDot(final Collection<NodeBase> nodes) { + final StringBuilder builder = new StringBuilder(); + builder.append("digraph {\n"); + for (final NodeBase node : nodes) { + final List<NodeBase> children = node.getChildren(); + + String style = ""; + if (node instanceof Decision) { + style = "[style=dashed];"; + } + + for (final NodeBase child : children) { + final String s = String.format("\t\"%s\" -> \"%s\"%s%n", node.getName(), child.getName(), style); + builder.append(s); + } + } + + builder.append("}"); + + return builder.toString(); + } + + private static String workflowToDot(final Workflow workflow) { + return nodesToDot(workflow.getNodes()); + } + + private static String nodesToDot(final Collection<Node> nodes) { + final StringBuilder builder = new StringBuilder(); + builder.append("digraph {\n"); + for (final Node node : nodes) { + final List<Node> children = node.getAllChildren(); + + String style = ""; + if (!node.getChildrenWithConditions().isEmpty()) { + style = "[style=dashed];"; + } + + for (final Node child : children) { + builder.append(String.format("\t\"%s\" -> \"%s\"%s%n", node.getName(), child.getName(), style)); + } + } + + builder.append("}"); + + return builder.toString(); + } + + public static void graphToPng(final Graph graph, final String fileName) throws IOException { + final MutableGraph mg = Parser.read(graphToDot(graph)); + mg.setLabel(fileName); + + Graphviz.fromGraph(mg) + .width(PNG_WIDTH) + .render(Format.PNG) + .toFile(new File(PARENT_FOLDER_NAME, FilenameUtils.getName(fileName))); + } + + public static void workflowToPng(final Workflow workflow, final String fileName) throws IOException { + final MutableGraph mg = Parser.read(workflowToDot(workflow)); + mg.setLabel(fileName); + + Graphviz.fromGraph(mg) + .width(PNG_WIDTH) + .render(Format.PNG) + .toFile(new File(PARENT_FOLDER_NAME, FilenameUtils.getName(fileName))); + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java ---------------------------------------------------------------------- diff --git a/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java new file mode 100644 index 0000000..9561372 --- /dev/null +++ b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.fluentjob.api; + +import com.google.common.base.Preconditions; + +/** + * A generic wrapper class for a value that can be modified once after construction, but only once. + * @param <T> the generic type that can be modified once, but only once + */ +public class ModifyOnce<T> { + private T data; + private boolean modified; + + /** + * Creates a new {@link ModifyOnce} object initialized to {@code null}. + */ + public ModifyOnce() { + this(null); + } + + /** + * Creates a new {@link ModifyOnce} object initialized to {@code defaultData}. + * @param defaultData The initial value of this {@link ModifyOnce} object. + */ + public ModifyOnce(final T defaultData) { + this.data = defaultData; + this.modified = false; + } + + /** + * Returns the wrapped value. + * @return The wrapped value. + */ + public T get() { + return data; + } + + /** + * Sets the wrapped value. If it is not the first modification attempt, {@link IllegalStateException} is thrown. + * @param data The new data to store. + * + * @throws IllegalStateException if this is not the first modification attempt. + */ + public void set(final T data) { + Preconditions.checkState(!modified, "Has already been modified once."); + + this.data = data; + this.modified = true; + } +} http://git-wip-us.apache.org/repos/asf/oozie/blob/8a0a6487/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java ---------------------------------------------------------------------- diff --git a/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java new file mode 100644 index 0000000..692eafb --- /dev/null +++ b/fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java @@ -0,0 +1,258 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.oozie.fluentjob.api.action; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +import java.util.List; +import java.util.Map; + +/** + * An immutable class holding data that is used by several actions. It should be constructed by using an + * {@link ActionAttributesBuilder}. + */ [email protected] [email protected] +public class ActionAttributes { + private final String resourceManager; + private final String nameNode; + private final Prepare prepare; + private final Streaming streaming; + private final Pipes pipes; + private final ImmutableList<String> jobXmls; + private final ImmutableMap<String, String> configuration; + private final String configClass; + private final ImmutableList<String> files; + private final ImmutableList<String> archives; + private final ImmutableList<Delete> deletes; + private final ImmutableList<Mkdir> mkdirs; + private final ImmutableList<Move> moves; + private final ImmutableList<Chmod> chmods; + private final ImmutableList<Touchz> touchzs; + private final ImmutableList<Chgrp> chgrps; + private final String javaOpts; + private final ImmutableList<String> args; + private final Launcher launcher; + private final Boolean captureOutput; + + ActionAttributes(final String resourceManager, + final String nameNode, + final Prepare prepare, + final Streaming streaming, + final Pipes pipes, + final ImmutableList<String> jobXmls, + final ImmutableMap<String, String> configuration, + final String configClass, + final ImmutableList<String> files, + final ImmutableList<String> archives, + final ImmutableList<Delete> deletes, + final ImmutableList<Mkdir> mkdirs, + final ImmutableList<Move> moves, + final ImmutableList<Chmod> chmods, + final ImmutableList<Touchz> touchzs, + final ImmutableList<Chgrp> chgrps, + final String javaOpts, + final ImmutableList<String> args, + final Launcher launcher, + final Boolean captureOutput) { + this.resourceManager = resourceManager; + this.nameNode = nameNode; + this.prepare = prepare; + this.streaming = streaming; + this.pipes = pipes; + this.jobXmls = jobXmls; + this.configuration = configuration; + this.configClass = configClass; + this.files = files; + this.archives = archives; + this.deletes = deletes; + this.mkdirs = mkdirs; + this.moves = moves; + this.chmods = chmods; + this.touchzs = touchzs; + this.chgrps = chgrps; + this.javaOpts = javaOpts; + this.args = args; + this.launcher = launcher; + this.captureOutput = captureOutput; + } + + /** + * Returns the resource manager address + * @return the resource manager address + */ + public String getResourceManager() { + return resourceManager; + } + + /** + * Returns the name node stored in this {@link ActionAttributes} object. + * @return The name node stored in this {@link ActionAttributes} object. + */ + public String getNameNode() { + return nameNode; + } + + /** + * Returns the {@link Prepare} object stored in this {@link ActionAttributes} object. + * @return The {@link Prepare} object stored in this {@link ActionAttributes} object. + */ + public Prepare getPrepare() { + return prepare; + } + + /** + * Returns the {@link Streaming} object stored in this {@link ActionAttributes} object. + * @return The {@link Streaming} object stored in this {@link ActionAttributes} object. + */ + public Streaming getStreaming() { + return streaming; + } + + /** + * Returns the {@link Pipes} object stored in this {@link ActionAttributes} object. + * @return The {@link Pipes} object stored in this {@link ActionAttributes} object. + */ + public Pipes getPipes() { + return pipes; + } + + /** + * Returns the list of job XMLs stored in this {@link ActionAttributes} object. + * @return The list of job XMLs stored in this {@link ActionAttributes} object. + */ + public List<String> getJobXmls() { + return jobXmls; + } + + /** + * Returns a map of the configuration key-value pairs stored in this {@link ActionAttributes} object. + * @return A map of the configuration key-value pairs stored in this {@link ActionAttributes} object. + */ + public Map<String, String> getConfiguration() { + return configuration; + } + + /** + * Returns the configuration class property of this {@link ActionAttributes} object. + * @return The configuration class property of this {@link ActionAttributes} object. + */ + public String getConfigClass() { + return configClass; + } + + /** + * Returns a list of the names of the files associated with this {@link ActionAttributes} object. + * @return A list of the names of the files associated with this {@link ActionAttributes} object. + */ + public List<String> getFiles() { + return files; + } + + /** + * Returns a list of the names of the archives associated with this {@link ActionAttributes} object. + * @return A list of the names of the archives associated with this {@link ActionAttributes} object. + */ + public List<String> getArchives() { + return archives; + } + + /** + * Returns a list of the {@link Delete} objects stored in this {@link ActionAttributes} object. + * @return A list of the {@link Delete} objects stored in this {@link ActionAttributes} object. + */ + public List<Delete> getDeletes() { + return deletes; + } + + /** + * Returns a list of the {@link Mkdir} objects stored in this {@link ActionAttributes} object. + * @return A list of the {@link Mkdir} objects stored in this {@link ActionAttributes} object. + */ + public List<Mkdir> getMkdirs() { + return mkdirs; + } + + /** + * Returns a list of the {@link Move} objects stored in this {@link ActionAttributes} object. + * @return A list of the {@link Move} objects stored in this {@link ActionAttributes} object. + */ + public List<Move> getMoves() { + return moves; + } + + /** + * Returns a list of the {@link Chmod} objects stored in this {@link ActionAttributes} object. + * @return A list of the {@link Chmod} objects stored in this {@link ActionAttributes} object. + */ + public List<Chmod> getChmods() { + return chmods; + } + + /** + * Returns a list of the {@link Touchz} objects stored in this {@link ActionAttributes} object. + * @return A list of the {@link Touchz} objects stored in this {@link ActionAttributes} object. + */ + public List<Touchz> getTouchzs() { + return touchzs; + } + + /** + * Returns a list of the {@link Chgrp} objects stored in this {@link ActionAttributes} object. + * @return A list of the {@link Delete} objects stored in this {@link ActionAttributes} object. + */ + public List<Chgrp> getChgrps() { + return chgrps; + } + + /** + * Get the java options. + * @return the java options + */ + public String getJavaOpts() { + return javaOpts; + } + + /** + * Get all the arguments. + * @return the argument list + */ + public List<String> getArgs() { + return args; + } + + /** + * Get the {@link Launcher} + * @return the {@link Launcher} + */ + public Launcher getLauncher() { + return launcher; + } + + /** + * Tells the caller whether to capture output or not. + * @return {@code true} when capturing output + */ + public boolean isCaptureOutput() { + return captureOutput == null ? false : captureOutput; + } +}
