Added: websites/staging/oozie/trunk/content/docs/5.1.0/WorkflowFunctionalSpec.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.1.0/WorkflowFunctionalSpec.html (added) +++ websites/staging/oozie/trunk/content/docs/5.1.0/WorkflowFunctionalSpec.html Wed Dec 19 15:50:03 2018 @@ -0,0 +1,4914 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p><hr /> +<h1>Oozie Specification, a Hadoop Workflow System</h1> +<p><b><center>(v5.0)</center></b></p> +<p>The goal of this document is to define a workflow engine system specialized in coordinating the execution of Hadoop Map/Reduce and Pig jobs.</p> +<ul> +<li><a href="#Changelog">Changelog</a></li> +<li><a href="#a0_Definitions">0 Definitions</a></li> +<li><a href="#a1_Specification_Highlights">1 Specification Highlights</a></li> +<li><a href="#a2_Workflow_Definition">2 Workflow Definition</a> +<ul> +<li><a href="#a2.1_Cycles_in_Workflow_Definitions">2.1 Cycles in Workflow Definitions</a></li></ul></li> +<li><a href="#a3_Workflow_Nodes">3 Workflow Nodes</a> +<ul> +<li><a href="#a3.1_Control_Flow_Nodes">3.1 Control Flow Nodes</a> +<ul> +<li><a href="#a3.1.1_Start_Control_Node">3.1.1 Start Control Node</a></li> +<li><a href="#a3.1.2_End_Control_Node">3.1.2 End Control Node</a></li> +<li><a href="#a3.1.3_Kill_Control_Node">3.1.3 Kill Control Node</a></li> +<li><a href="#a3.1.4_Decision_Control_Node">3.1.4 Decision Control Node</a></li> +<li><a href="#a3.1.5_Fork_and_Join_Control_Nodes">3.1.5 Fork and Join Control Nodes</a></li></ul></li> +<li><a href="#a3.2_Workflow_Action_Nodes">3.2 Workflow Action Nodes</a> +<ul> +<li><a href="#a3.2.1_Action_Basis">3.2.1 Action Basis</a> +<ul> +<li><a href="#a3.2.1.1_Action_ComputationProcessing_Is_Always_Remote">3.2.1.1 Action Computation/Processing Is Always Remote</a></li> +<li><a href="#a3.2.1.2_Actions_Are_Asynchronous">3.2.1.2 Actions Are Asynchronous</a></li> +<li><a href="#a3.2.1.3_Actions_Have_2_Transitions_ok_and_error">3.2.1.3 Actions Have 2 Transitions, ok and error</a></li> +<li><a href="#a3.2.1.4_Action_Recovery">3.2.1.4 Action Recovery</a></li></ul></li> +<li><a href="#a3.2.2_Map-Reduce_Action">3.2.2 Map-Reduce Action</a> +<ul> +<li><a href="#a3.2.2.1_Adding_Files_and_Archives_for_the_Job">3.2.2.1 Adding Files and Archives for the Job</a></li> +<li><a href="#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code">3.2.2.2 Configuring the MapReduce action with Java code</a></li> +<li><a href="#a3.2.2.3_Streaming">3.2.2.3 Streaming</a></li> +<li><a href="#a3.2.2.4_Pipes">3.2.2.4 Pipes</a></li> +<li><a href="#a3.2.2.5_Syntax">3.2.2.5 Syntax</a></li></ul></li> +<li><a href="#a3.2.3_Pig_Action">3.2.3 Pig Action</a></li> +<li><a href="#a3.2.4_Fs_HDFS_action">3.2.4 Fs (HDFS) action</a></li> +<li><a href="#a3.2.5_Sub-workflow_Action">3.2.5 Sub-workflow Action</a></li> +<li><a href="#a3.2.6_Java_Action">3.2.6 Java Action</a> +<ul> +<li><a href="#a3.2.6.1_Overriding_an_actions_Main_class">3.2.6.1 Overriding an action’s Main class</a></li></ul></li></ul></li></ul></li> +<li><a href="#a4_Parameterization_of_Workflows">4 Parameterization of Workflows</a> +<ul> +<li><a href="#a4.1_Workflow_Job_Properties_or_Parameters">4.1 Workflow Job Properties (or Parameters)</a></li> +<li><a href="#a4.2_Expression_Language_Functions">4.2 Expression Language Functions</a> +<ul> +<li><a href="#a4.2.1_Basic_EL_Constants">4.2.1 Basic EL Constants</a></li> +<li><a href="#a4.2.2_Basic_EL_Functions">4.2.2 Basic EL Functions</a></li> +<li><a href="#a4.2.3_Workflow_EL_Functions">4.2.3 Workflow EL Functions</a></li> +<li><a href="#a4.2.4_Hadoop_EL_Constants">4.2.4 Hadoop EL Constants</a></li> +<li><a href="#a4.2.5_Hadoop_EL_Functions">4.2.5 Hadoop EL Functions</a></li> +<li><a href="#a4.2.6_Hadoop_Jobs_EL_Function">4.2.6 Hadoop Jobs EL Function</a></li> +<li><a href="#a4.2.7_HDFS_EL_Functions">4.2.7 HDFS EL Functions</a></li> +<li><a href="#a4.2.8_HCatalog_EL_Functions">4.2.8 HCatalog EL Functions</a></li></ul></li></ul></li> +<li><a href="#a5_Workflow_Notifications">5 Workflow Notifications</a> +<ul> +<li><a href="#a5.1_Workflow_Job_Status_Notification">5.1 Workflow Job Status Notification</a></li> +<li><a href="#a5.2_Node_Start_and_End_Notifications">5.2 Node Start and End Notifications</a></li></ul></li> +<li><a href="#a6_User_Propagation">6 User Propagation</a></li> +<li><a href="#a7_Workflow_Application_Deployment">7 Workflow Application Deployment</a></li> +<li><a href="#a8_External_Data_Assumptions">8 External Data Assumptions</a></li> +<li><a href="#a9_Workflow_Jobs_Lifecycle">9 Workflow Jobs Lifecycle</a> +<ul> +<li><a href="#a9.1_Workflow_Job_Lifecycle">9.1 Workflow Job Lifecycle</a></li> +<li><a href="#a9.2_Workflow_Action_Lifecycle">9.2 Workflow Action Lifecycle</a></li></ul></li> +<li><a href="#a10_Workflow_Jobs_Recovery_re-run">10 Workflow Jobs Recovery (re-run)</a></li> +<li><a href="#a11_Oozie_Web_Services_API">11 Oozie Web Services API</a></li> +<li><a href="#a12_Client_API">12 Client API</a></li> +<li><a href="#a13_Command_Line_Tools">13 Command Line Tools</a></li> +<li><a href="#a14_Web_UI_Console">14 Web UI Console</a></li> +<li><a href="#a15_Customizing_Oozie_with_Extensions">15 Customizing Oozie with Extensions</a></li> +<li><a href="#a16_Workflow_Jobs_Priority">16 Workflow Jobs Priority</a></li> +<li><a href="#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3">17 HDFS Share Libraries for Workflow Applications (since Oozie 2.3)</a> +<ul> +<li><a href="#a17.1_Action_Share_Library_Override_since_Oozie_3.3">17.1 Action Share Library Override (since Oozie 3.3)</a></li></ul></li> +<li><a href="#a18_User-Retry_for_Workflow_Actions_since_Oozie_3.1">18 User-Retry for Workflow Actions (since Oozie 3.1)</a></li> +<li><a href="#a19_Global_Configurations">19 Global Configurations</a></li> +<li><a href="#a20_Suspend_On_Nodes">20 Suspend On Nodes</a></li> +<li><a href="#Appendixes">Appendixes</a> +<ul> +<li><a href="#Appendix_A_Oozie_Workflow_and_Common_XML_Schemas">Appendix A, Oozie Workflow and Common XML Schemas</a> +<ul> +<li><a href="#Oozie_Workflow_Schema_Version_1.0">Oozie Workflow Schema Version 1.0</a></li> +<li><a href="#Oozie_Common_Schema_Version_1.0">Oozie Common Schema Version 1.0</a></li> +<li><a href="#Oozie_Workflow_Schema_Version_0.5">Oozie Workflow Schema Version 0.5</a></li> +<li><a href="#Oozie_Workflow_Schema_Version_0.4.5">Oozie Workflow Schema Version 0.4.5</a></li> +<li><a href="#Oozie_Workflow_Schema_Version_0.4">Oozie Workflow Schema Version 0.4</a></li> +<li><a href="#Oozie_Workflow_Schema_Version_0.3">Oozie Workflow Schema Version 0.3</a></li> +<li><a href="#Oozie_Workflow_Schema_Version_0.2.5">Oozie Workflow Schema Version 0.2.5</a></li> +<li><a href="#Oozie_Workflow_Schema_Version_0.2">Oozie Workflow Schema Version 0.2</a></li> +<li><a href="#Oozie_SLA_Version_0.2">Oozie SLA Version 0.2</a></li> +<li><a href="#Oozie_SLA_Version_0.1">Oozie SLA Version 0.1</a></li> +<li><a href="#Oozie_Workflow_Schema_Version_0.1">Oozie Workflow Schema Version 0.1</a></li></ul></li> +<li><a href="#Appendix_B_Workflow_Examples">Appendix B, Workflow Examples</a> +<ul> +<li><a href="#Fork_and_Join_Example">Fork and Join Example</a></li></ul></li></ul></li></ul> + +<div class="section"> +<h2><a name="Changelog"></a>Changelog</h2> +<p><b>2016FEB19</b></p> +<ul> + +<li>3.2.7 Updated notes on System.exit(int n) behavior</li> +</ul> +<p><b>2015APR29</b></p> +<ul> + +<li>3.2.1.4 Added notes about Java action retries</li> +<li>3.2.7 Added notes about Java action retries</li> +</ul> +<p><b>2014MAY08</b></p> +<ul> + +<li>3.2.2.4 Added support for fully qualified job-xml path</li> +</ul> +<p><b>2013JUL03</b></p> +<ul> + +<li>Appendix A, Added new workflow schema 0.5 and SLA schema 0.2</li> +</ul> +<p><b>2012AUG30</b></p> +<ul> + +<li>4.2.2 Added two EL functions (replaceAll and appendAll)</li> +</ul> +<p><b>2012JUL26</b></p> +<ul> + +<li>Appendix A, updated XML schema 0.4 to include <tt>parameters</tt> element</li> +<li>4.1 Updated to mention about <tt>parameters</tt> element as of schema 0.4</li> +</ul> +<p><b>2012JUL23</b></p> +<ul> + +<li>Appendix A, updated XML schema 0.4 (Fs action)</li> +<li>3.2.4 Updated to mention that a <tt>name-node</tt>, a <tt>job-xml</tt>, and a <tt>configuration</tt> element are allowed in the Fs action as of schema 0.4</li> +</ul> +<p><b>2012JUN19</b></p> +<ul> + +<li>Appendix A, added XML schema 0.4</li> +<li>3.2.2.4 Updated to mention that multiple <tt>job-xml</tt> elements are allowed as of schema 0.4</li> +<li>3.2.3 Updated to mention that multiple <tt>job-xml</tt> elements are allowed as of schema 0.4</li> +</ul> +<p><b>2011AUG17</b></p> +<ul> + +<li>3.2.4 fs ‘chmod’ xml closing element typo in Example corrected</li> +</ul> +<p><b>2011AUG12</b></p> +<ul> + +<li>3.2.4 fs ‘move’ action characteristics updated, to allow for consistent source and target paths and existing target path only if directory</li> +<li>18, Update the doc for user-retry of workflow action.</li> +</ul> +<p><b>2011FEB19</b></p> +<ul> + +<li>10, Update the doc to rerun from the failed node.</li> +</ul> +<p><b>2010OCT31</b></p> +<ul> + +<li>17, Added new section on Shared Libraries</li> +</ul> +<p><b>2010APR27</b></p> +<ul> + +<li>3.2.3 Added new “arguments” tag to PIG actions</li> +<li>3.2.5 SSH actions are deprecated in Oozie schema 0.1 and removed in Oozie schema 0.2</li> +<li>Appendix A, Added schema version 0.2</li> +</ul> +<p><b>2009OCT20</b></p> +<ul> + +<li>Appendix A, updated XML schema</li> +</ul> +<p><b>2009SEP15</b></p> +<ul> + +<li>3.2.6 Removing support for sub-workflow in a different Oozie instance (removing the ‘oozie’ element)</li> +</ul> +<p><b>2009SEP07</b></p> +<ul> + +<li>3.2.2.3 Added Map Reduce Pipes specifications.</li> +<li>3.2.2.4 Map-Reduce Examples. Previously was 3.2.2.3.</li> +</ul> +<p><b>2009SEP02</b></p> +<ul> + +<li>10 Added missing skip nodes property name.</li> +<li>3.2.1.4 Reworded action recovery explanation.</li> +</ul> +<p><b>2009AUG26</b></p> +<ul> + +<li>3.2.9 Added <tt>java</tt> action type</li> +<li>3.1.4 Example uses EL constant to refer to counter group/name</li> +</ul> +<p><b>2009JUN09</b></p> +<ul> + +<li>12.2.4 Added build version resource to admin end-point</li> +<li>3.2.6 Added flag to propagate workflow configuration to sub-workflows</li> +<li>10 Added behavior for workflow job parameters given in the rerun</li> +<li>11.3.4 workflows info returns pagination information</li> +</ul> +<p><b>2009MAY18</b></p> +<ul> + +<li>3.1.4 decision node, ‘default’ element, ‘name’ attribute changed to ‘to’</li> +<li>3.1.5 fork node, ‘transition’ element changed to ‘start’, ‘to’ attribute change to ‘path’</li> +<li>3.1.5 join node, ‘transition’ element remove, added ‘to’ attribute to ‘join’ element</li> +<li>3.2.1.4 Rewording on action recovery section</li> +<li>3.2.2 map-reduce action, added ‘job-tracker’, ‘name-node’ actions, ‘file’, ‘file’ and ‘archive’ elements</li> +<li>3.2.2.1 map-reduce action, remove from ‘streaming’ element ‘file’, ‘file’ and ‘archive’ elements</li> +<li>3.2.2.2 map-reduce action, reorganized streaming section</li> +<li>3.2.3 pig action, removed information about implementation (SSH), changed elements names</li> +<li>3.2.4 fs action, removed ‘fs-uri’ and ‘user-name’ elements, file system URI is now specified in path, user is propagated</li> +<li>3.2.6 sub-workflow action, renamed elements ‘oozie-url’ to ‘oozie’ and ‘workflow-app’ to ‘app-path’</li> +<li>4 Properties that are valid Java identifiers can be used as ${NAME}</li> +<li>4.1 Renamed default properties file from ‘configuration.xml’ to ‘default-configuration.xml’</li> +<li>4.2 Changes in EL Constants and Functions</li> +<li>5 Updated notification behavior and tokens</li> +<li>6 Changed user propagation behavior</li> +<li>7 Changed application packaging from ZIP to HDFS directory</li> +<li>Removed application lifecycle and self containment model sections</li> +<li>10 Changed workflow job recovery, simplified recovery behavior</li> +<li>11 Detailed Web Services API</li> +<li>12 Updated Client API section</li> +<li>15 Updated Action Executor API section</li> +<li>Appendix A XML namespace updated to ‘uri:oozie:workflow:0.1’</li> +<li>Appendix A Updated XML schema to changes in map-reduce/pig/fs/ssh actions</li> +<li>Appendix B Updated workflow example to schema changes</li> +</ul> +<p><b>2009MAR25</b></p> +<ul> + +<li>Changing all references of HWS to Oozie (project name)</li> +<li>Typos, XML Formatting</li> +<li>XML Schema URI correction</li> +</ul> +<p><b>2009MAR09</b></p> +<ul> + +<li>Changed <tt>CREATED</tt> job state to <tt>PREP</tt> to have same states as Hadoop</li> +<li>Renamed ‘hadoop-workflow’ element to ‘workflow-app’</li> +<li>Decision syntax changed to be ‘switch/case’ with no transition indirection</li> +<li>Action nodes common root element ‘action’, with the action type as sub-element (using a single built-in XML schema)</li> +<li>Action nodes have 2 explicit transitions ‘ok to’ and ‘error to’ enforced by XML schema</li> +<li>Renamed ‘fail’ action element to ‘kill’</li> +<li>Renamed ‘hadoop’ action element to ‘map-reduce’</li> +<li>Renamed ‘hdfs’ action element to ‘fs’</li> +<li>Updated all XML snippets and examples</li> +<li>Made user propagation simpler and consistent</li> +<li>Added Oozie XML schema to Appendix A</li> +<li>Added workflow example to Appendix B</li> +</ul> +<p><b>2009FEB22</b></p> +<ul> + +<li>Opened <a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-5303";>JIRA HADOOP-5303</a></li> +</ul> +<p><b>27/DEC/2012:</b></p> +<ul> + +<li>Added information on dropping hcatalog table partitions in prepare block</li> +<li>Added hcatalog EL functions section</li> +</ul></div> +<div class="section"> +<h2><a name="a0_Definitions"></a>0 Definitions</h2> +<p><b>Action:</b> An execution/computation task (Map-Reduce job, Pig job, a shell command). It can also be referred as task or ‘action node’.</p> +<p><b>Workflow:</b> A collection of actions arranged in a control dependency DAG (Directed Acyclic Graph). “control dependency” from one action to another means that the second action can’t run until the first action has completed.</p> +<p><b>Workflow Definition:</b> A programmatic description of a workflow that can be executed.</p> +<p><b>Workflow Definition Language:</b> The language used to define a Workflow Definition.</p> +<p><b>Workflow Job:</b> An executable instance of a workflow definition.</p> +<p><b>Workflow Engine:</b> A system that executes workflows jobs. It can also be referred as a DAG engine.</p></div> +<div class="section"> +<h2><a name="a1_Specification_Highlights"></a>1 Specification Highlights</h2> +<p>A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, and sub-workflows.</p> +<p>Flow control operations within the workflow applications can be done using decision, fork and join nodes. Cycles in workflows are not supported.</p> +<p>Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow definition as <tt>${VAR}</tt> variables.</p> +<p>A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig scripts, and other resource files.</p> +<p>Before running a workflow job, the corresponding workflow application must be deployed in Oozie.</p> +<p>Deploying workflow application and running workflow jobs can be done via command line tools, a WS API and a Java API.</p> +<p>Monitoring the system and workflow jobs can be done via a web console, command line tools, a WS API and a Java API.</p> +<p>When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions must be provided. This set of properties is a Hadoop configuration.</p> +<p>Possible states for a workflow jobs are: <tt>PREP</tt>, <tt>RUNNING</tt>, <tt>SUSPENDED</tt>, <tt>SUCCEEDED</tt>, <tt>KILLED</tt> and <tt>FAILED</tt>.</p> +<p>In the case of a action start failure in a workflow job, depending on the type of failure, Oozie will attempt automatic retries, it will request a manual retry or it will fail the workflow job.</p> +<p>Oozie can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.</p> +<p>In the case of workflow job failure, the workflow job can be resubmitted skipping previously completed actions. Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow application code.</p> +<p><a name="WorkflowDefinition"></a></p></div> +<div class="section"> +<h2><a name="a2_Workflow_Definition"></a>2 Workflow Definition</h2> +<p>A workflow definition is a DAG with control flow nodes (start, end, decision, fork, join, kill) or action nodes (map-reduce, pig, etc.), nodes are connected by transitions arrows.</p> +<p>The workflow definition language is XML based and it is called hPDL (Hadoop Process Definition Language).</p> +<p>Refer to the Appendix A for the<a href="WorkflowFunctionalSpec.html#OozieWFSchema">Oozie Workflow Definition XML Schema</a>. Appendix B has <a href="WorkflowFunctionalSpec.html#OozieWFExamples">Workflow Definition Examples</a>.</p> +<div class="section"> +<h3><a name="a2.1_Cycles_in_Workflow_Definitions"></a>2.1 Cycles in Workflow Definitions</h3> +<p>Oozie does not support cycles in workflow definitions, workflow definitions must be a strict DAG.</p> +<p>At workflow application deployment time, if Oozie detects a cycle in the workflow definition it must fail the deployment.</p></div></div> +<div class="section"> +<h2><a name="a3_Workflow_Nodes"></a>3 Workflow Nodes</h2> +<p>Workflow nodes are classified in control flow nodes and action nodes:</p> +<ul> + +<li><b>Control flow nodes:</b> nodes that control the start and end of the workflow and workflow job execution path.</li> +<li><b>Action nodes:</b> nodes that trigger the execution of a computation/processing task.</li> +</ul> +<p>Node names and transitions must be conform to the following pattern <tt>[a-zA-Z][\-_a-zA-Z0-0]*</tt>, of up to 20 characters long.</p> +<div class="section"> +<h3><a name="a3.1_Control_Flow_Nodes"></a>3.1 Control Flow Nodes</h3> +<p>Control flow nodes define the beginning and the end of a workflow (the <tt>start</tt>, <tt>end</tt> and <tt>kill</tt> nodes) and provide a mechanism to control the workflow execution path (the <tt>decision</tt>, <tt>fork</tt> and <tt>join</tt> nodes).</p> +<p><a name="StartNode"></a></p> +<div class="section"> +<h4><a name="a3.1.1_Start_Control_Node"></a>3.1.1 Start Control Node</h4> +<p>The <tt>start</tt> node is the entry point for a workflow job, it indicates the first workflow node the workflow job must transition to.</p> +<p>When a workflow is started, it automatically transitions to the node specified in the <tt>start</tt>.</p> +<p>A workflow definition must have one <tt>start</tt> node.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <start to="[NODE-NAME]"/> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>to</tt> attribute is the name of first workflow node to execute.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <start to="firstHadoopJob"/> + ... +</workflow-app> +</pre></div></div> + +<p><a name="EndNode"></a></p></div> +<div class="section"> +<h4><a name="a3.1.2_End_Control_Node"></a>3.1.2 End Control Node</h4> +<p>The <tt>end</tt> node is the end for a workflow job, it indicates that the workflow job has completed successfully.</p> +<p>When a workflow job reaches the <tt>end</tt> it finishes successfully (SUCCEEDED).</p> +<p>If one or more actions started by the workflow job are executing when the <tt>end</tt> node is reached, the actions will be killed. In this scenario the workflow job is still considered as successfully run.</p> +<p>A workflow definition must have one <tt>end</tt> node.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <end name="[NODE-NAME]"/> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>name</tt> attribute is the name of the transition to do to end the workflow job.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <end name="end"/> +</workflow-app> +</pre></div></div> + +<p><a name="KillNode"></a></p></div> +<div class="section"> +<h4><a name="a3.1.3_Kill_Control_Node"></a>3.1.3 Kill Control Node</h4> +<p>The <tt>kill</tt> node allows a workflow job to kill itself.</p> +<p>When a workflow job reaches the <tt>kill</tt> it finishes in error (KILLED).</p> +<p>If one or more actions started by the workflow job are executing when the <tt>kill</tt> node is reached, the actions will be killed.</p> +<p>A workflow definition may have zero or more <tt>kill</tt> nodes.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <kill name="[NODE-NAME]"> + <message>[MESSAGE-TO-LOG]</message> + </kill> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>name</tt> attribute in the <tt>kill</tt> node is the name of the Kill action node.</p> +<p>The content of the <tt>message</tt> element will be logged as the kill reason for the workflow job.</p> +<p>A <tt>kill</tt> node does not have transition elements because it ends the workflow job, as <tt>KILLED</tt>.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <kill name="killBecauseNoInput"> + <message>Input unavailable</message> + </kill> + ... +</workflow-app> +</pre></div></div> + +<p><a name="DecisionNode"></a></p></div> +<div class="section"> +<h4><a name="a3.1.4_Decision_Control_Node"></a>3.1.4 Decision Control Node</h4> +<p>A <tt>decision</tt> node enables a workflow to make a selection on the execution path to follow.</p> +<p>The behavior of a <tt>decision</tt> node can be seen as a switch-case statement.</p> +<p>A <tt>decision</tt> node consists of a list of predicates-transition pairs plus a default transition. Predicates are evaluated in order or appearance until one of them evaluates to <tt>true</tt> and the corresponding transition is taken. If none of the predicates evaluates to <tt>true</tt> the <tt>default</tt> transition is taken.</p> +<p>Predicates are JSP Expression Language (EL) expressions (refer to section 4.2 of this document) that resolve into a boolean value, <tt>true</tt> or <tt>false</tt>. For example:</p> + +<div> +<div> +<pre class="source"> ${fs:fileSize('/usr/foo/myinputdir') gt 10 * GB} +</pre></div></div> + +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <decision name="[NODE-NAME]"> + <switch> + <case to="[NODE_NAME]">[PREDICATE]</case> + ... + <case to="[NODE_NAME]">[PREDICATE]</case> + <default to="[NODE_NAME]"/> + </switch> + </decision> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>name</tt> attribute in the <tt>decision</tt> node is the name of the decision node.</p> +<p>Each <tt>case</tt> elements contains a predicate and a transition name. The predicate ELs are evaluated in order until one returns <tt>true</tt> and the corresponding transition is taken.</p> +<p>The <tt>default</tt> element indicates the transition to take if none of the predicates evaluates to <tt>true</tt>.</p> +<p>All decision nodes must have a <tt>default</tt> element to avoid bringing the workflow into an error state if none of the predicates evaluates to true.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <decision name="mydecision"> + <switch> + <case to="reconsolidatejob"> + ${fs:fileSize(secondjobOutputDir) gt 10 * GB} + </case> <case to="rexpandjob"> + ${fs:fileSize(secondjobOutputDir) lt 100 * MB} + </case> + <case to="recomputejob"> + ${ hadoop:counters('secondjob')[RECORDS][REDUCE_OUT] lt 1000000 } + </case> + <default to="end"/> + </switch> + </decision> + ... +</workflow-app> +</pre></div></div> + +<p><a name="ForkJoinNodes"></a></p></div> +<div class="section"> +<h4><a name="a3.1.5_Fork_and_Join_Control_Nodes"></a>3.1.5 Fork and Join Control Nodes</h4> +<p>A <tt>fork</tt> node splits one path of execution into multiple concurrent paths of execution.</p> +<p>A <tt>join</tt> node waits until every concurrent execution path of a previous <tt>fork</tt> node arrives to it.</p> +<p>The <tt>fork</tt> and <tt>join</tt> nodes must be used in pairs. The <tt>join</tt> node assumes concurrent execution paths are children of the same <tt>fork</tt> node.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <fork name="[FORK-NODE-NAME]"> + <path start="[NODE-NAME]" /> + ... + <path start="[NODE-NAME]" /> + </fork> + ... + <join name="[JOIN-NODE-NAME]" to="[NODE-NAME]" /> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>name</tt> attribute in the <tt>fork</tt> node is the name of the workflow fork node. The <tt>start</tt> attribute in the <tt>path</tt> elements in the <tt>fork</tt> node indicate the name of the workflow node that will be part of the concurrent execution paths.</p> +<p>The <tt>name</tt> attribute in the <tt>join</tt> node is the name of the workflow join node. The <tt>to</tt> attribute in the <tt>join</tt> node indicates the name of the workflow node that will executed after all concurrent execution paths of the corresponding fork arrive to the join node.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <fork name="forking"> + <path start="firstparalleljob"/> + <path start="secondparalleljob"/> + </fork> + <action name="firstparallejob"> + <map-reduce> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <job-xml>job1.xml</job-xml> + </map-reduce> + <ok to="joining"/> + <error to="kill"/> + </action> + <action name="secondparalleljob"> + <map-reduce> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <job-xml>job2.xml</job-xml> + </map-reduce> + <ok to="joining"/> + <error to="kill"/> + </action> + <join name="joining" to="nextaction"/> + ... +</workflow-app> +</pre></div></div> + +<p>By default, Oozie performs some validation that any forking in a workflow is valid and won’t lead to any incorrect behavior or instability. However, if Oozie is preventing a workflow from being submitted and you are very certain that it should work, you can disable forkjoin validation so that Oozie will accept the workflow. To disable this validation just for a specific workflow, simply set <tt>oozie.wf.validate.ForkJoin</tt> to <tt>false</tt> in the job.properties file. To disable this validation for all workflows, simply set <tt>oozie.validate.ForkJoin</tt> to <tt>false</tt> in the oozie-site.xml file. Disabling this validation is determined by the AND of both of these properties, so it will be disabled if either or both are set to false and only enabled if both are set to true (or not specified).</p> +<p><a name="ActionNodes"></a></p></div></div> +<div class="section"> +<h3><a name="a3.2_Workflow_Action_Nodes"></a>3.2 Workflow Action Nodes</h3> +<p>Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task.</p> +<div class="section"> +<h4><a name="a3.2.1_Action_Basis"></a>3.2.1 Action Basis</h4> +<p>The following sub-sections define common behavior and capabilities for all action types.</p> +<div class="section"> +<h5><a name="a3.2.1.1_Action_ComputationProcessing_Is_Always_Remote"></a>3.2.1.1 Action Computation/Processing Is Always Remote</h5> +<p>All computation/processing tasks triggered by an action node are remote to Oozie. No workflow application specific computation/processing task is executed within Oozie.</p></div> +<div class="section"> +<h5><a name="a3.2.1.2_Actions_Are_Asynchronous"></a>3.2.1.2 Actions Are Asynchronous</h5> +<p>All computation/processing tasks triggered by an action node are executed asynchronously by Oozie. For most types of computation/processing tasks triggered by workflow action, the workflow job has to wait until the computation/processing task completes before transitioning to the following node in the workflow.</p> +<p>The exception is the <tt>fs</tt> action that is handled as a synchronous action.</p> +<p>Oozie can detect completion of computation/processing tasks by two different means, callbacks and polling.</p> +<p>When a computation/processing tasks is started by Oozie, Oozie provides a unique callback URL to the task, the task should invoke the given URL to notify its completion.</p> +<p>For cases that the task failed to invoke the callback URL for any reason (i.e. a transient network failure) or when the type of task cannot invoke the callback URL upon completion, Oozie has a mechanism to poll computation/processing tasks for completion.</p></div> +<div class="section"> +<h5><a name="a3.2.1.3_Actions_Have_2_Transitions_ok_and_error"></a>3.2.1.3 Actions Have 2 Transitions, <tt>ok</tt> and <tt>error</tt></h5> +<p>If a computation/processing task -triggered by a workflow- completes successfully, it transitions to <tt>ok</tt>.</p> +<p>If a computation/processing task -triggered by a workflow- fails to complete successfully, its transitions to <tt>error</tt>.</p> +<p>If a computation/processing task exits in error, there computation/processing task must provide <tt>error-code</tt> and <tt>error-message</tt> information to Oozie. This information can be used from <tt>decision</tt> nodes to implement a fine grain error handling at workflow application level.</p> +<p>Each action type must clearly define all the error codes it can produce.</p></div> +<div class="section"> +<h5><a name="a3.2.1.4_Action_Recovery"></a>3.2.1.4 Action Recovery</h5> +<p>Oozie provides recovery capabilities when starting or ending actions.</p> +<p>Once an action starts successfully Oozie will not retry starting the action if the action fails during its execution. The assumption is that the external system (i.e. Hadoop) executing the action has enough resilience to recover jobs once it has started (i.e. Hadoop task retries).</p> +<p>Java actions are a special case with regard to retries. Although Oozie itself does not retry Java actions should they fail after they have successfully started, Hadoop itself can cause the action to be restarted due to a map task retry on the map task running the Java application. See the Java Action section below for more detail.</p> +<p>For failures that occur prior to the start of the job, Oozie will have different recovery strategies depending on the nature of the failure.</p> +<p>If the failure is of transient nature, Oozie will perform retries after a pre-defined time interval. The number of retries and timer interval for a type of action must be pre-configured at Oozie level. Workflow jobs can override such configuration.</p> +<p>Examples of a transient failures are network problems or a remote system temporary unavailable.</p> +<p>If the failure is of non-transient nature, Oozie will suspend the workflow job until an manual or programmatic intervention resumes the workflow job and the action start or end is retried. It is the responsibility of an administrator or an external managing system to perform any necessary cleanup before resuming the workflow job.</p> +<p>If the failure is an error and a retry will not resolve the problem, Oozie will perform the error transition for the action.</p> +<p><a name="MapReduceAction"></a></p></div></div> +<div class="section"> +<h4><a name="a3.2.2_Map-Reduce_Action"></a>3.2.2 Map-Reduce Action</h4> +<p>The <tt>map-reduce</tt> action starts a Hadoop map/reduce job from a workflow. Hadoop jobs can be Java Map/Reduce jobs or streaming jobs.</p> +<p>A <tt>map-reduce</tt> action can be configured to perform file system cleanup and directory creation before starting the map reduce job. This capability enables Oozie to retry a Hadoop job in the situation of a transient failure (Hadoop checks the non-existence of the job output directory and then creates it when the Hadoop job is starting, thus a retry without cleanup of the job output directory would fail).</p> +<p>The workflow job will wait until the Hadoop map/reduce job completes before continuing to the next action in the workflow execution path.</p> +<p>The counters of the Hadoop job and job exit status (<tt>FAILED</tt>, <tt>KILLED</tt> or <tt>SUCCEEDED</tt>) must be available to the workflow job after the Hadoop jobs ends. This information can be used from within decision nodes and other actions configurations.</p> +<p>The <tt>map-reduce</tt> action has to be configured with all the necessary Hadoop JobConf properties to run the Hadoop map/reduce job.</p> +<p>Hadoop JobConf properties can be specified as part of</p> +<ul> + +<li>the <tt>config-default.xml</tt> or</li> +<li>JobConf XML file bundled with the workflow application or</li> +<li><global> tag in workflow definition or</li> +<li>Inline <tt>map-reduce</tt> action configuration or</li> +<li>An implementation of OozieActionConfigurator specified by the <config-class> tag in workflow definition.</li> +</ul> +<p>The configuration properties are loaded in the following above order i.e. <tt>streaming</tt>, <tt>job-xml</tt>, <tt>configuration</tt>, and <tt>config-class</tt>, and the precedence order is later values override earlier values.</p> +<p>Streaming and inline property values can be parameterized (templatized) using EL expressions.</p> +<p>The Hadoop <tt>mapred.job.tracker</tt> and <tt>fs.default.name</tt> properties must not be present in the job-xml and inline configuration.</p> +<p><a name="FilesArchives"></a></p> +<div class="section"> +<h5><a name="a3.2.2.1_Adding_Files_and_Archives_for_the_Job"></a>3.2.2.1 Adding Files and Archives for the Job</h5> +<p>The <tt>file</tt>, <tt>archive</tt> elements make available, to map-reduce jobs, files and archives. If the specified path is relative, it is assumed the file or archiver are within the application directory, in the corresponding sub-path. If the path is absolute, the file or archive it is expected in the given absolute path.</p> +<p>Files specified with the <tt>file</tt> element, will be symbolic links in the home directory of the task.</p> +<p>If a file is a native library (an ‘.so’ or a ‘.so.#’ file), it will be symlinked as and ‘.so’ file in the task running directory, thus available to the task JVM.</p> +<p>To force a symlink for a file on the task running directory, use a ‘#’ followed by the symlink name. For example ‘mycat.sh#cat’.</p> +<p>Refer to Hadoop distributed cache documentation for details more details on files and archives.</p></div> +<div class="section"> +<h5><a name="a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code"></a>3.2.2.2 Configuring the MapReduce action with Java code</h5> +<p>Java code can be used to further configure the MapReduce action. This can be useful if you already have “driver” code for your MapReduce action, if you’re more familiar with MapReduce’s Java API, if there’s some configuration that requires logic, or some configuration that’s difficult to do in straight XML (e.g. Avro).</p> +<p>Create a class that implements the org.apache.oozie.action.hadoop.OozieActionConfigurator interface from the “oozie-sharelib-oozie” artifact. It contains a single method that receives a <tt>JobConf</tt> as an argument. Any configuration properties set on this <tt>JobConf</tt> will be used by the MapReduce action.</p> +<p>The OozieActionConfigurator has this signature:</p> + +<div> +<div> +<pre class="source">public interface OozieActionConfigurator { + public void configure(JobConf actionConf) throws OozieActionConfiguratorException; +} +</pre></div></div> + +<p>where <tt>actionConf</tt> is the <tt>JobConf</tt> you can update. If you need to throw an Exception, you can wrap it in an <tt>OozieActionConfiguratorException</tt>, also in the “oozie-sharelib-oozie” artifact.</p> +<p>For example:</p> + +<div> +<div> +<pre class="source">package com.example; + +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.mapred.FileInputFormat; +import org.apache.hadoop.mapred.FileOutputFormat; +import org.apache.hadoop.mapred.JobConf; +import org.apache.oozie.action.hadoop.OozieActionConfigurator; +import org.apache.oozie.action.hadoop.OozieActionConfiguratorException; +import org.apache.oozie.example.SampleMapper; +import org.apache.oozie.example.SampleReducer; + +public class MyConfigClass implements OozieActionConfigurator { + + @Override + public void configure(JobConf actionConf) throws OozieActionConfiguratorException { + if (actionConf.getUser() == null) { + throw new OozieActionConfiguratorException("No user set"); + } + actionConf.setMapperClass(SampleMapper.class); + actionConf.setReducerClass(SampleReducer.class); + FileInputFormat.setInputPaths(actionConf, new Path("/user/" + actionConf.getUser() + "/input-data")); + FileOutputFormat.setOutputPath(actionConf, new Path("/user/" + actionConf.getUser() + "/output")); + ... + } +} +</pre></div></div> + +<p>To use your config class in your MapReduce action, simply compile it into a jar, make the jar available to your action, and specify the class name in the <tt>config-class</tt> element (this requires at least schema 0.5):</p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <map-reduce> + ... + <job-xml>[JOB-XML-FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <config-class>com.example.MyConfigClass</config-class> + ... + </map-reduce> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>Another example of this can be found in the “map-reduce” example that comes with Oozie.</p> +<p>A useful tip: The initial <tt>JobConf</tt> passed to the <tt>configure</tt> method includes all of the properties listed in the <tt>configuration</tt> section of the MR action in a workflow. If you need to pass any information to your OozieActionConfigurator, you can simply put them here.</p> +<p><a name="StreamingMapReduceAction"></a></p></div> +<div class="section"> +<h5><a name="a3.2.2.3_Streaming"></a>3.2.2.3 Streaming</h5> +<p>Streaming information can be specified in the <tt>streaming</tt> element.</p> +<p>The <tt>mapper</tt> and <tt>reducer</tt> elements are used to specify the executable/script to be used as mapper and reducer.</p> +<p>User defined scripts must be bundled with the workflow application and they must be declared in the <tt>files</tt> element of the streaming configuration. If the are not declared in the <tt>files</tt> element of the configuration it is assumed they will be available (and in the command PATH) of the Hadoop slave machines.</p> +<p>Some streaming jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using the <tt>file</tt> and <tt>archive</tt> elements described in the previous section.</p> +<p>The Mapper/Reducer can be overridden by a <tt>mapred.mapper.class</tt> or <tt>mapred.reducer.class</tt> properties in the <tt>job-xml</tt> file or <tt>configuration</tt> elements.</p> +<p><a name="PipesMapReduceAction"></a></p></div> +<div class="section"> +<h5><a name="a3.2.2.4_Pipes"></a>3.2.2.4 Pipes</h5> +<p>Pipes information can be specified in the <tt>pipes</tt> element.</p> +<p>A subset of the command line options which can be used while using the Hadoop Pipes Submitter can be specified via elements - <tt>map</tt>, <tt>reduce</tt>, <tt>inputformat</tt>, <tt>partitioner</tt>, <tt>writer</tt>, <tt>program</tt>.</p> +<p>The <tt>program</tt> element is used to specify the executable/script to be used.</p> +<p>User defined program must be bundled with the workflow application.</p> +<p>Some pipe jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using the <tt>file</tt> and <tt>archive</tt> elements described in the previous section.</p> +<p>Pipe properties can be overridden by specifying them in the <tt>job-xml</tt> file or <tt>configuration</tt> element.</p></div> +<div class="section"> +<h5><a name="a3.2.2.5_Syntax"></a>3.2.2.5 Syntax</h5> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <map-reduce> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <streaming> + <mapper>[MAPPER-PROCESS]</mapper> + <reducer>[REDUCER-PROCESS]</reducer> + <record-reader>[RECORD-READER-CLASS]</record-reader> + <record-reader-mapping>[NAME=VALUE]</record-reader-mapping> + ... + <env>[NAME=VALUE]</env> + ... + </streaming> + <!-- Either streaming or pipes can be specified for an action, not both --> + <pipes> + <map>[MAPPER]</map> + <reduce>[REDUCER]</reducer> + <inputformat>[INPUTFORMAT]</inputformat> + <partitioner>[PARTITIONER]</partitioner> + <writer>[OUTPUTFORMAT]</writer> + <program>[EXECUTABLE]</program> + </pipes> + <job-xml>[JOB-XML-FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <config-class>com.example.MyConfigClass</config-class> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </map-reduce> + + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>prepare</tt> element, if present, indicates a list of paths to delete before starting the job. This should be used exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation will be performed in the <tt>fs.default.name</tt> filesystem for hdfs URIs. The format for specifying hcatalog table URI is hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is <tt>hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]</tt>. In case of a hcatalog URI, the hive-site.xml needs to be shipped using <tt>file</tt> tag and the hcatalog and hive jars need to be placed in workflow lib directory or specified using <tt>archive</tt> tag.</p> +<p>The <tt>job-xml</tt> element, if present, must refer to a Hadoop JobConf <tt>job.xml</tt> file bundled in the workflow application. By default the <tt>job.xml</tt> file is taken from the workflow application namenode, regardless the namenode specified for the action. To specify a <tt>job.xml</tt> on another namenode use a fully qualified file path. The <tt>job-xml</tt> element is optional and as of schema 0.4, multiple <tt>job-xml</tt> elements are allowed in order to specify multiple Hadoop JobConf <tt>job.xml</tt> files.</p> +<p>The <tt>configuration</tt> element, if present, contains JobConf properties for the Hadoop job.</p> +<p>Properties specified in the <tt>configuration</tt> element override properties specified in the file specified in the <tt>job-xml</tt> element.</p> +<p>As of schema 0.5, the <tt>config-class</tt> element, if present, contains a class that implements OozieActionConfigurator that can be used to further configure the MapReduce job.</p> +<p>Properties specified in the <tt>config-class</tt> class override properties specified in <tt>configuration</tt> element.</p> +<p>External Stats can be turned on/off by specifying the property <i>oozie.action.external.stats.write</i> as <i>true</i> or <i>false</i> in the configuration element of workflow.xml. The default value for this property is <i>false</i>.</p> +<p>The <tt>file</tt> element, if present, must specify the target symbolic link for binaries by separating the original file and target with a # (file#target-sym-link). This is not required for libraries.</p> +<p>The <tt>mapper</tt> and <tt>reducer</tt> process for streaming jobs, should specify the executable command with URL encoding. e.g. ‘%’ should be replaced by ‘%25’.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myfirstHadoopJob"> + <map-reduce> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="hdfs://foo:8020/usr/tucu/output-data"/> + </prepare> + <job-xml>/myfirstjob.xml</job-xml> + <configuration> + <property> + <name>mapred.input.dir</name> + <value>/usr/tucu/input-data</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>/usr/tucu/input-data</value> + </property> + <property> + <name>mapred.reduce.tasks</name> + <value>${firstJobReducers}</value> + </property> + <property> + <name>oozie.action.external.stats.write</name> + <value>true</value> + </property> + </configuration> + </map-reduce> + <ok to="myNextAction"/> + <error to="errorCleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>In the above example, the number of Reducers to be used by the Map/Reduce job has to be specified as a parameter of the workflow job configuration when creating the workflow job.</p> +<p><b>Streaming Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="firstjob"> + <map-reduce> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${output}"/> + </prepare> + <streaming> + <mapper>/bin/bash testarchive/bin/mapper.sh testfile</mapper> + <reducer>/bin/bash testarchive/bin/reducer.sh</reducer> + </streaming> + <configuration> + <property> + <name>mapred.input.dir</name> + <value>${input}</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>${output}</value> + </property> + <property> + <name>stream.num.map.output.key.fields</name> + <value>3</value> + </property> + </configuration> + <file>/users/blabla/testfile.sh#testfile</file> + <archive>/users/blabla/testarchive.jar#testarchive</archive> + </map-reduce> + <ok to="end"/> + <error to="kill"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p><b>Pipes Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="firstjob"> + <map-reduce> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${output}"/> + </prepare> + <pipes> + <program>bin/wordcount-simple#wordcount-simple</program> + </pipes> + <configuration> + <property> + <name>mapred.input.dir</name> + <value>${input}</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>${output}</value> + </property> + </configuration> + <archive>/users/blabla/testarchive.jar#testarchive</archive> + </map-reduce> + <ok to="end"/> + <error to="kill"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p><a name="PigAction"></a></p></div></div> +<div class="section"> +<h4><a name="a3.2.3_Pig_Action"></a>3.2.3 Pig Action</h4> +<p>The <tt>pig</tt> action starts a Pig job.</p> +<p>The workflow job will wait until the pig job completes before continuing to the next action.</p> +<p>The <tt>pig</tt> action has to be configured with the resource-manager, name-node, pig script and the necessary parameters and configuration to run the Pig job.</p> +<p>A <tt>pig</tt> action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before starting the Pig job. This capability enables Oozie to retry a Pig job in the situation of a transient failure (Pig creates temporary directories for intermediate data, thus a retry without cleanup would fail).</p> +<p>Hadoop JobConf properties can be specified as part of</p> +<ul> + +<li>the <tt>config-default.xml</tt> or</li> +<li>JobConf XML file bundled with the workflow application or</li> +<li><global> tag in workflow definition or</li> +<li>Inline <tt>pig</tt> action configuration.</li> +</ul> +<p>The configuration properties are loaded in the following above order i.e. <tt>job-xml</tt> and <tt>configuration</tt>, and the precedence order is later values override earlier values.</p> +<p>Inline property values can be parameterized (templatized) using EL expressions.</p> +<p>The YARN <tt>yarn.resourcemanager.address</tt> and HDFS <tt>fs.default.name</tt> properties must not be present in the job-xml and inline configuration.</p> +<p>As with Hadoop map-reduce jobs, it is possible to add files and archives to be available to the Pig job, refer to section [#FilesArchives][Adding Files and Archives for the Job].</p> +<p><b>Syntax for Pig actions in Oozie schema 1.0:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <pig> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[JOB-XML-FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <script>[PIG-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <argument>[ARGUMENT-VALUE]</argument> + ... + <argument>[ARGUMENT-VALUE]</argument> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </pig> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p><b>Syntax for Pig actions in Oozie schema 0.2:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.2"> + ... + <action name="[NODE-NAME]"> + <pig> + <job-tracker>[JOB-TRACKER]</job-tracker> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[JOB-XML-FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <script>[PIG-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <argument>[ARGUMENT-VALUE]</argument> + ... + <argument>[ARGUMENT-VALUE]</argument> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </pig> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p><b>Syntax for Pig actions in Oozie schema 0.1:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> + ... + <action name="[NODE-NAME]"> + <pig> + <job-tracker>[JOB-TRACKER]</job-tracker> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[JOB-XML-FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <script>[PIG-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </pig> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>prepare</tt> element, if present, indicates a list of paths to delete before starting the job. This should be used exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation will be performed in the <tt>fs.default.name</tt> filesystem for hdfs URIs. The format for specifying hcatalog table URI is hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is <tt>hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]</tt>. In case of a hcatalog URI, the hive-site.xml needs to be shipped using <tt>file</tt> tag and the hcatalog and hive jars need to be placed in workflow lib directory or specified using <tt>archive</tt> tag.</p> +<p>The <tt>job-xml</tt> element, if present, must refer to a Hadoop JobConf <tt>job.xml</tt> file bundled in the workflow application. The <tt>job-xml</tt> element is optional and as of schema 0.4, multiple <tt>job-xml</tt> elements are allowed in order to specify multiple Hadoop JobConf <tt>job.xml</tt> files.</p> +<p>The <tt>configuration</tt> element, if present, contains JobConf properties for the underlying Hadoop jobs.</p> +<p>Properties specified in the <tt>configuration</tt> element override properties specified in the file specified in the <tt>job-xml</tt> element.</p> +<p>External Stats can be turned on/off by specifying the property <i>oozie.action.external.stats.write</i> as <i>true</i> or <i>false</i> in the configuration element of workflow.xml. The default value for this property is <i>false</i>.</p> +<p>The inline and job-xml configuration properties are passed to the Hadoop jobs submitted by Pig runtime.</p> +<p>The <tt>script</tt> element contains the pig script to execute. The pig script can be templatized with variables of the form <tt>${VARIABLE}</tt>. The values of these variables can then be specified using the <tt>params</tt> element.</p> +<p>NOTE: Oozie will perform the parameter substitution before firing the pig job. This is different from the <a class="externalLink" href="http://wiki.apache.org/pig/ParameterSubstitution";>parameter substitution mechanism provided by Pig</a>, which has a few limitations.</p> +<p>The <tt>params</tt> element, if present, contains parameters to be passed to the pig script.</p> +<p><b>In Oozie schema 0.2:</b> The <tt>arguments</tt> element, if present, contains arguments to be passed to the pig script.</p> +<p>All the above elements can be parameterized (templatized) using EL expressions.</p> +<p><b>Example for Oozie schema 0.2:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.2"> + ... + <action name="myfirstpigjob"> + <pig> + <job-tracker>foo:8021</job-tracker> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + <property> + <name>oozie.action.external.stats.write</name> + <value>true</value> + </property> + </configuration> + <script>/mypigscript.pig</script> + <argument>-param</argument> + <argument>INPUT=${inputDir}</argument> + <argument>-param</argument> + <argument>OUTPUT=${outputDir}/pig-output3</argument> + </pig> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p><b>Example for Oozie schema 0.1:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> + ... + <action name="myfirstpigjob"> + <pig> + <job-tracker>foo:8021</job-tracker> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <script>/mypigscript.pig</script> + <param>InputDir=/home/tucu/input-data</param> + <param>OutputDir=${jobOutput}</param> + </pig> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p><a name="FsAction"></a></p></div> +<div class="section"> +<h4><a name="a3.2.4_Fs_HDFS_action"></a>3.2.4 Fs (HDFS) action</h4> +<p>The <tt>fs</tt> action allows to manipulate files and directories in HDFS from a workflow application. The supported commands are <tt>move</tt>, <tt>delete</tt>, <tt>mkdir</tt>, <tt>chmod</tt>, <tt>touchz</tt>, <tt>setrep</tt> and <tt>chgrp</tt>.</p> +<p>The FS commands are executed synchronously from within the FS action, the workflow job will wait until the specified file commands are completed before continuing to the next action.</p> +<p>Path names specified in the <tt>fs</tt> action can be parameterized (templatized) using EL expressions. Path name should be specified as a absolute path. In case of <tt>move</tt>, <tt>delete</tt>, <tt>chmod</tt> and <tt>chgrp</tt> commands, a glob pattern can also be specified instead of an absolute path. For <tt>move</tt>, glob pattern can only be specified for source path and not the target.</p> +<p>Each file path must specify the file system URI, for move operations, the target must not specify the system URI.</p> +<p><b>IMPORTANT:</b> For the purposes of copying files within a cluster it is recommended to refer to the <tt>distcp</tt> action instead. Refer to <a href="DG_DistCpActionExtension.html"><tt>distcp</tt></a> action to copy files within a cluster.</p> +<p><b>IMPORTANT:</b> All the commands within <tt>fs</tt> action do not happen atomically, if a <tt>fs</tt> action fails half way in the commands being executed, successfully executed commands are not rolled back. The <tt>fs</tt> action, before executing any command must check that source paths exist and target paths don’t exist (constraint regarding target relaxed for the <tt>move</tt> action. See below for details), thus failing before executing any command. Therefore the validity of all paths specified in one <tt>fs</tt> action are evaluated before any of the file operation are executed. Thus there is less chance of an error occurring while the <tt>fs</tt> action executes.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <fs> + <delete path='[PATH]' skip-trash='[true/false]'/> + ... + <mkdir path='[PATH]'/> + ... + <move source='[SOURCE-PATH]' target='[TARGET-PATH]'/> + ... + <chmod path='[PATH]' permissions='[PERMISSIONS]' dir-files='false' /> + ... + <touchz path='[PATH]' /> + ... + <chgrp path='[PATH]' group='[GROUP]' dir-files='false' /> + ... + <setrep path='[PATH]' replication-factor='2'/> + </fs> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>delete</tt> command deletes the specified path, if it is a directory it deletes recursively all its content and then deletes the directory. By default it does skip trash. It can be moved to trash by setting the value of skip-trash to ‘false’. It can also be used to drop hcat tables/partitions. This is the only FS command which supports HCatalog URIs as well. For eg:</p> + +<div> +<div> +<pre class="source"><delete path='hcat://[metastore server]:[port]/[database name]/[table name]'/> +OR +<delete path='hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...'/> +</pre></div></div> + +<p>The <tt>mkdir</tt> command creates the specified directory, it creates all missing directories in the path. If the directory already exist it does a no-op.</p> +<p>In the <tt>move</tt> command the <tt>source</tt> path must exist. The following scenarios are addressed for a <tt>move</tt>:</p> +<ul> + +<li>The file system URI(e.g. <tt>hdfs://{nameNode}</tt>) can be skipped in the <tt>target</tt> path. It is understood to be the same as that of the source. But if the target path does contain the system URI, it cannot be different than that of the source.</li> +<li>The parent directory of the <tt>target</tt> path must exist</li> +<li>For the <tt>target</tt> path, if it is a file, then it must not already exist.</li> +<li>However, if the <tt>target</tt> path is an already existing directory, the <tt>move</tt> action will place your <tt>source</tt> as a child of the <tt>target</tt> directory.</li> +</ul> +<p>The <tt>chmod</tt> command changes the permissions for the specified path. Permissions can be specified using the Unix Symbolic representation (e.g. -rwxrw-rw-) or an octal representation (755). When doing a <tt>chmod</tt> command on a directory, by default the command is applied to the directory and the files one level within the directory. To apply the <tt>chmod</tt> command to the directory, without affecting the files within it, the <tt>dir-files</tt> attribute must be set to <tt>false</tt>. To apply the <tt>chmod</tt> command recursively to all levels within a directory, put a <tt>recursive</tt> element inside the <chmod> element.</p> +<p>The <tt>touchz</tt> command creates a zero length file in the specified path if none exists. If one already exists, then touchz will perform a touch operation. Touchz works only for absolute paths.</p> +<p>The <tt>chgrp</tt> command changes the group for the specified path. When doing a <tt>chgrp</tt> command on a directory, by default the command is applied to the directory and the files one level within the directory. To apply the <tt>chgrp</tt> command to the directory, without affecting the files within it, the <tt>dir-files</tt> attribute must be set to <tt>false</tt>. To apply the <tt>chgrp</tt> command recursively to all levels within a directory, put a <tt>recursive</tt> element inside the <chgrp> element.</p> +<p>The <tt>setrep</tt> command changes replication factor of an hdfs file(s). Changing RF of directories or symlinks is not supported; this action requires an argument for RF.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="hdfscommands"> + <fs> + <delete path='hdfs://foo:8020/usr/tucu/temp-data'/> + <mkdir path='archives/${wf:id()}'/> + <move source='${jobInput}' target='archives/${wf:id()}/processed-input'/> + <chmod path='${jobOutput}' permissions='-rwxrw-rw-' dir-files='true'><recursive/></chmod> + <chgrp path='${jobOutput}' group='testgroup' dir-files='true'><recursive/></chgrp> + <setrep path='archives/${wf:id()/filename(s)}' replication-factor='2'/> + </fs> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>In the above example, a directory named after the workflow job ID is created and the input of the job, passed as workflow configuration parameter, is archived under the previously created directory.</p> +<p>As of schema 0.4, if a <tt>name-node</tt> element is specified, then it is not necessary for any of the paths to start with the file system URI as it is taken from the <tt>name-node</tt> element. This is also true if the name-node is specified in the global section (see <a href="WorkflowFunctionalSpec.html#GlobalConfigurations">Global Configurations</a>)</p> +<p>As of schema 0.4, zero or more <tt>job-xml</tt> elements can be specified; these must refer to Hadoop JobConf <tt>job.xml</tt> formatted files bundled in the workflow application. They can be used to set additional properties for the FileSystem instance.</p> +<p>As of schema 0.4, if a <tt>configuration</tt> element is specified, then it will also be used to set additional JobConf properties for the FileSystem instance. Properties specified in the <tt>configuration</tt> element override properties specified in the files specified by any <tt>job-xml</tt> elements.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.4"> + ... + <action name="hdfscommands"> + <fs> + <name-node>hdfs://foo:8020</name-node> + <job-xml>fs-info.xml</job-xml> + <configuration> + <property> + <name>some.property</name> + <value>some.value</value> + </property> + </configuration> + <delete path='/usr/tucu/temp-data'/> + </fs> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p><a name="SubWorkflowAction"></a></p></div> +<div class="section"> +<h4><a name="a3.2.5_Sub-workflow_Action"></a>3.2.5 Sub-workflow Action</h4> +<p>The <tt>sub-workflow</tt> action runs a child workflow job, the child workflow job can be in the same Oozie system or in another Oozie system.</p> +<p>The parent workflow job will wait until the child workflow job has completed.</p> +<p>There can be several sub-workflows defined within a single workflow, each under its own action element.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <sub-workflow> + <app-path>[WF-APPLICATION-PATH]</app-path> + <propagate-configuration/> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + </sub-workflow> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The child workflow job runs in the same Oozie system instance where the parent workflow job is running.</p> +<p>The <tt>app-path</tt> element specifies the path to the workflow application of the child workflow job.</p> +<p>The <tt>propagate-configuration</tt> flag, if present, indicates that the workflow job configuration should be propagated to the child workflow.</p> +<p>The <tt>configuration</tt> section can be used to specify the job properties that are required to run the child workflow job.</p> +<p>The configuration of the <tt>sub-workflow</tt> action can be parameterized (templatized) using EL expressions.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="a"> + <sub-workflow> + <app-path>child-wf</app-path> + <configuration> + <property> + <name>input.dir</name> + <value>${wf:id()}/second-mr-output</value> + </property> + </configuration> + </sub-workflow> + <ok to="end"/> + <error to="kill"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>In the above example, the workflow definition with the name <tt>child-wf</tt> will be run on the Oozie instance at <tt>.http://myhost:11000/oozie</tt>. The specified workflow application must be already deployed on the target Oozie instance.</p> +<p>A configuration parameter <tt>input.dir</tt> is being passed as job property to the child workflow job.</p> +<p>The subworkflow can inherit the lib jars from the parent workflow by setting <tt>oozie.subworkflow.classpath.inheritance</tt> to true in oozie-site.xml or on a per-job basis by setting <tt>oozie.wf.subworkflow.classpath.inheritance</tt> to true in a job.properties file. If both are specified, <tt>oozie.wf.subworkflow.classpath.inheritance</tt> has priority. If the subworkflow and the parent have conflicting jars, the subworkflow’s jar has priority. By default, <tt>oozie.wf.subworkflow.classpath.inheritance</tt> is set to false.</p> +<p>To prevent errant workflows from starting infinitely recursive subworkflows, <tt>oozie.action.subworkflow.max.depth</tt> can be specified in oozie-site.xml to set the maximum depth of subworkflow calls. For example, if set to 3, then a workflow can start subwf1, which can start subwf2, which can start subwf3; but if subwf3 tries to start subwf4, then the action will fail. The default is 50.</p> +<p><a name="JavaAction"></a></p></div> +<div class="section"> +<h4><a name="a3.2.6_Java_Action"></a>3.2.6 Java Action</h4> +<p>The <tt>java</tt> action will execute the <tt>public static void main(String[] args)</tt> method of the specified main Java class.</p> +<p>Java applications are executed in the Hadoop cluster as map-reduce job with a single Mapper task.</p> +<p>The workflow job will wait until the java application completes its execution before continuing to the next action.</p> +<p>The <tt>java</tt> action has to be configured with the resource-manager, name-node, main Java class, JVM options and arguments.</p> +<p>To indicate an <tt>ok</tt> action transition, the main Java class must complete gracefully the <tt>main</tt> method invocation.</p> +<p>To indicate an <tt>error</tt> action transition, the main Java class must throw an exception.</p> +<p>The main Java class can call <tt>System.exit(int n)</tt>. Exit code zero is regarded as OK, while non-zero exit codes will cause the <tt>java</tt> action to do an <tt>error</tt> transition and exit.</p> +<p>A <tt>java</tt> action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before starting the Java application. This capability enables Oozie to retry a Java application in the situation of a transient or non-transient failure (This can be used to cleanup any temporary data which may have been created by the Java application in case of failure).</p> +<p>A <tt>java</tt> action can create a Hadoop configuration for interacting with a cluster (e.g. launching a map-reduce job). Oozie prepares a Hadoop configuration file which includes the environments site configuration files (e.g. hdfs-site.xml, mapred-site.xml, etc) plus the properties added to the <tt><configuration></tt> section of the <tt>java</tt> action. The Hadoop configuration file is made available as a local file to the Java application in its running directory. It can be added to the <tt>java</tt> actions Hadoop configuration by referencing the system property: <tt>oozie.action.conf.xml</tt>. For example:</p> + +<div> +<div> +<pre class="source">// loading action conf prepared by Oozie +Configuration actionConf = new Configuration(false); +actionConf.addResource(new Path("file:///", System.getProperty("oozie.action.conf.xml"))); +</pre></div></div> + +<p>If <tt>oozie.action.conf.xml</tt> is not added then the job will pick up the mapred-default properties and this may result in unexpected behaviour. For repeated configuration properties later values override earlier ones.</p> +<p>Inline property values can be parameterized (templatized) using EL expressions.</p> +<p>The YARN <tt>yarn.resourcemanager.address</tt> (<tt>resource-manager</tt>) and HDFS <tt>fs.default.name</tt> (<tt>name-node</tt>) properties must not be present in the <tt>job-xml</tt> and in the inline configuration.</p> +<p>As with <tt>map-reduce</tt> and <tt>pig</tt> actions, it is possible to add files and archives to be available to the Java application. Refer to section [#FilesArchives][Adding Files and Archives for the Job].</p> +<p>The <tt>capture-output</tt> element can be used to propagate values back into Oozie context, which can then be accessed via EL-functions. This needs to be written out as a java properties format file. The filename is obtained via a System property specified by the constant <tt>oozie.action.output.properties</tt></p> +<p><b>IMPORTANT:</b> In order for a Java action to succeed on a secure cluster, it must propagate the Hadoop delegation token like in the following code snippet (this is benign on non-secure clusters):</p> + +<div> +<div> +<pre class="source">// propagate delegation related props from launcher job to MR job +if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { + jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION")); +} +</pre></div></div> + +<p><b>IMPORTANT:</b> Because the Java application is run from within a Map-Reduce job, from Hadoop 0.20. onwards a queue must be assigned to it. The queue name must be specified as a configuration property.</p> +<p><b>IMPORTANT:</b> The Java application from a Java action is executed in a single map task. If the task is abnormally terminated, such as due to a TaskTracker restart (e.g. during cluster maintenance), the task will be retried via the normal Hadoop task retry mechanism. To avoid workflow failure, the application should be written in a fashion that is resilient to such retries, for example by detecting and deleting incomplete outputs or picking back up from complete outputs. Furthermore, if a Java action spawns asynchronous activity outside the JVM of the action itself (such as by launching additional MapReduce jobs), the application must consider the possibility of collisions with activity spawned by the new instance.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <java> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[JOB-XML]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <main-class>[MAIN-CLASS]</main-class> + <java-opts>[JAVA-STARTUP-OPTS]</java-opts> + <arg>ARGUMENT</arg> + ... + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + <capture-output /> + </java> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>prepare</tt> element, if present, indicates a list of paths to delete before starting the Java application. This should be used exclusively for directory cleanup or dropping of hcatalog table or table partitions for the Java application to be executed. In case of <tt>delete</tt>, a glob pattern can be used to specify path. The format for specifying hcatalog table URI is hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is <tt>hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]</tt>. In case of a hcatalog URI, the hive-site.xml needs to be shipped using <tt>file</tt> tag and the hcatalog and hive jars need to be placed in workflow lib directory or specified using <tt>archive</tt> tag.</p>
[... 3480 lines stripped ...] Added: websites/staging/oozie/trunk/content/docs/5.1.0/css/apache-maven-fluido-1.4.min.css ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.1.0/css/apache-maven-fluido-1.4.min.css (added) +++ websites/staging/oozie/trunk/content/docs/5.1.0/css/apache-maven-fluido-1.4.min.css Wed Dec 19 15:50:03 2018 @@ -0,0 +1,9 @@ +/*! + * Bootstrap v2.3.2 + * + * Copyright 2013 Twitter, Inc + * Licensed under the Apache License v2.0 + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Designed and built with all the love in the world by @mdo and @fat. [... 3 lines stripped ...]
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}