Added: oozie/site/trunk/content/resources/docs/5.0.0/DG_CoordinatorRerun.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/DG_CoordinatorRerun.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/DG_CoordinatorRerun.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/DG_CoordinatorRerun.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,184 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="Coordinator_Rerun"></a> +<div class="section"><h2> Coordinator Rerun</h2> +<p><ul><ul><li><a href="#Pre-Conditions">Pre-Conditions</a> +</li> +<li><a href="#Rerun_Arguments">Rerun Arguments</a> +</li> +<li><a href="#Rerun_coordinator_actions">Rerun coordinator actions</a> +</li> +</ul> +</ul> +</p> +<a name="Pre-Conditions"></a> +<div class="section"><h3>Pre-Conditions</h3> +<p><ul><li>Rerun coordinator action must be in TIMEDOUT/SUCCEEDED/KILLED/FAILED.</li> +<li>Coordinator actions cannot be rerun if the coordinator job is in the PREP or IGNORED state.</li> +<li>Rerun a PAUSED coordinator job, the status and pause time and pending flag will not be reset.</li> +<li>Rerun a SUSPENDED coordinator job, the status will reset to RUNNING.</li> +<li>All rerun actions must exist already.</li> +<li>Coordinator Rerun will only use the original configs from first run.</li> +<li>Coordinator Rerun will not re-read the coordinator.xml in hdfs.</li> +</ul> +</p> +<a name="Rerun_Arguments"></a> +</div> +<div class="section"><h3>Rerun Arguments</h3> +<p><pre> +$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>] +[-action 1, 3-4, 7-40] (-action or -date is required to rerun.) +[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] +(if neither -action nor -date is given, the exception will be thrown.) +</pre></p> +<p><ul><li>Either -action or -date should be given.</li> +<li>If -action and -date both are given, an error will be thrown.</li> +<li>Multiple ranges can be used in -action or -date. See the above examples.</li> +<li>If one of action in the given list of -action does not exist or not in terminal state, the rerun throws an error.</li> +<li>The dates specified in -date must be UTC.</li> +<li>Single date specified in -date must be able to find an action with matched nominal time to be effective.</li> +<li>If -nocleanup is given, coordinator directories will not be removed; otherwise the 'output-event' will be deleted, unless nocleanup attribute is explicitly set in coordinator.xml</li> +<li>If -refresh is set, new dataset is re-evaluated for latest() and future().</li> +<li>If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked.</li> +<li>If -failed is set, re-runs the failed workflow actions of the coordinator actions.</li> +<li>-config can be used to supply properties to workflow by job configuration file '.xml' or '.properties'.</li> +</ul> +</p> +<a name="Rerun_coordinator_actions"></a> +</div> +<div class="section"><h3>Rerun coordinator actions</h3> +<p><ul><li>Rerun terminated (timeout, succeeded, killed, failed) coordinator actions.</li> +<li>By default, Oozie will delete the 'output-event' directories before changing actions' status and materializing actions.</li> +<li>If coordinator job is RUNNING, rerun actions will be materialized and compete with current running actions.</li> +<li>Rerun for job, user should use job's start date and end date in -date.</li> +<li>If the user specifies a date range (say Jan 1 to May 1), the actions that will be re-run are the existing actions</li> +</ul> +within that range. If the existing actions are action #5....#40, which map to Jan 15 to Feb 15, then only those actions will run.<ul><li>The rerun action_id and nominal_time of the actions which are eligible to rerun will be returned.</li> +</ul> +</p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
Added: oozie/site/trunk/content/resources/docs/5.0.0/DG_CustomActionExecutor.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/DG_CustomActionExecutor.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/DG_CustomActionExecutor.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/DG_CustomActionExecutor.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,233 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="Custom_Action_Nodes"></a> +<div class="section"><h2> Custom Action Nodes</h2> +<p><ul><ul><li><a href="#Introduction">Introduction</a> +</li> +<li><a href="#Writing_a_custom_Action_Node">Writing a custom Action Node</a> +<ul><li><a href="#ActionExecutor.Context">ActionExecutor.Context</a> +</li> +<li><a href="#Constructor">Constructor</a> +</li> +<li><a href="#initActionType">initActionType()</a> +</li> +<li><a href="#startActionExecutor.Context_context_Action_action">start(ActionExecutor.Context context, Action action)</a> +</li> +<li><a href="#checkActionExecutor.Context_context_Action_action">check(ActionExecutor.Context context, Action action)</a> +</li> +<li><a href="#killActionExecutor.Context_context_Action_action">kill(ActionExecutor.Context context, Action action)</a> +</li> +<li><a href="#endActionExecutor.Context_context_Action_action">end(ActionExecutor.Context context, Action action)</a> +</li> +<li><a href="#Registering_Errors">Registering Errors</a> +</li> +</ul> +</li> +<li><a href="#Deploying_a_custom_Action_Executor">Deploying a custom Action Executor</a> +</li> +</ul> +</ul> +</p> +<a name="Introduction"></a> +<div class="section"><h3>Introduction</h3> +<p>Oozie can be extended to support additional action types by writing a custom <a href="./WorkflowFunctionalSpec.html#ActionNodes">Action Node</a> +. Action Nodes can be synchronous or asynchronous.<ul><li>Synchronous Node - Sync nodes are executed inline by Oozie, which waits for completion of these nodes before proceeding. Hence, these nodes should almost never be used and are meant for lightweight tasks like FileSystem move, mkdir, delete.</li> +<li>Asynchronous Nodes - Oozie starts asynchronous nodes, and then monitors the action being executed for completion. This is done via a callback from the action or Oozie polling for the action status.</li> +</ul> +</p> +<a name="Writing_a_custom_Action_Node"></a> +</div> +<div class="section"><h3>Writing a custom Action Node</h3> +<p>Action Executors are configured in the oozie configuration file oozie-site.xml. These executors are loaded during Oozie startup. <a href="./DG_CustomActionExecutor.html#Deploying_a_custom_Action_Executor">Deploying a Custom Action Executor</a> +.</p> +<p>Action Executors MUST extend the <tt>ActionExecutor</tt> + class and override the required methods.</p> +<p>Most methods take as argument the Execution Context and the actual Action object with various configuration properties resolved.</p> +<a name="ActionExecutor.Context"></a> +<div class="section"><h4>ActionExecutor.Context</h4> +<p>The Execution context gives Action Nodes access to configuration properties, methods to set the state of the action, methods to set variables which are to be made available later in the execution path.</p> +<p><b>The following methods from the ActionExecutor interface should be implemented.</b> +</p> +<a name="Constructor"></a> +</div> +<div class="section"><h4>Constructor</h4> +<p>A no argument constructor should be implemented, which calls super(ACTION_TYPE). ACTION_TYPE is the name of the action which will be used in the workflow xml, and is used by Oozie to instantiate the correct type of Executor.</p> +<a name="initActionType"></a> +</div> +<div class="section"><h4>initActionType()</h4> +<p>This method is called once during initialization of the Action Executor during Oozie startup. Any common initialization code for the Action Node should go here.</p> +<p>As an example, setting up of error handling for the Custom Action should be done here.</p> +<p>This method must call super.initActionType() as it's first statement.</p> +<a name="startActionExecutor.Context_context_Action_action"></a> +</div> +<div class="section"><h4>start(ActionExecutor.Context context, Action action)</h4> +<p>The action start up happens here.<ul><li>Async Actions - The action should be started and context.setStartData(externalId, trackerUri, consoleUrl) must be set. A check can be made for whether the action has completed, in which case context.setExecutionData(externalStatus, actionData) must be called.</li> +<li>Sync Actions - The action should be started and should complete execution. context.setExecutionData(externalStatus, actionData) must be called.</li> +</ul> +</p> +<a name="checkActionExecutor.Context_context_Action_action"></a> +</div> +<div class="section"><h4>check(ActionExecutor.Context context, Action action)</h4> +<p><tt>check(...)</tt> + is used by Oozie to poll for the status of the action. This method should interact with the action started previously, and update the status. If the action has completed, context.setExecutionData(externalStatus, actionData) must be called. Otherwise, the status can be updated using context.setExternalStatus(externalStatus).</p> +<p>For sync actions, this method will not be called, and should throw an UnsupportedOperationException().</p> +<a name="killActionExecutor.Context_context_Action_action"></a> +</div> +<div class="section"><h4>kill(ActionExecutor.Context context, Action action)</h4> +<p><tt>kill(...)</tt> + is called when there is an attempt to kill the running job or action. No workflow transition is made after this.</p> +<p>The implementation for a custom action should interact with and kill the running action, and take care of any cleanup which may be required. context.setEndData(status, signalValue) should be called with both values set to Action.Status.KILLED.</p> +<a name="endActionExecutor.Context_context_Action_action"></a> +</div> +<div class="section"><h4>end(ActionExecutor.Context context, Action action)</h4> +<p><tt>end(...)</end> is used for any cleanup or processing which may need to be done after completion of the action. After any processing, context.setEndData(status, signalValue) should be called to complete execution of the action and trigger the next workflow transition. signalValue can be Action.Status.OK or Action.Status.ERROR.</p> +<a name="Registering_Errors"></a> +</div> +<div class="section"><h4>Registering Errors</h4> +<p>Oozie actions can generate different types of Errors.<ul><li>TRANSIENT - will be retried</li> +<li>NON TRANSIENT - the job will be suspended and can be resumed later by human intervention, after fixing whatever problem caused this error.</li> +<li>ERROR - causes the error transition to be taken.</li> +<li>FAILED - the action and the job are set to FAILED state. No transitions are taken.</li> +</ul> +registerError(exceptionClassName, errorType, errorMessage) can be used to register possible exceptions while executing the action, along with their type and error message. This will normally be done during initialization of the Action Executor.</p> +<a name="Deploying_a_custom_Action_Executor"></a> +</div> +</div> +<div class="section"><h3>Deploying a custom Action Executor</h3> +<p>Action Nodes can be registered in the oozie configuration file oozie-site.xml, by changing the property 'oozie.service.ActionService.executor.ext.classes'. For multiple Executors, the class name should be separated by commas. +<pre> <property> + <name>oozie.service.ActionService.executor.ext.classes</name> + <value> + org.apache.oozie.wf.action.decision.CustomActionExecutor, + Custom_Action_Executr_2.class + </value> + </property></pre> +Any configuration properties to be made available to this class should also be added to oozie-site.xml. The convention to be followed for naming these properties is 'oozie.action.[ActionName].property.name'</p> +<p>The XML schema (XSD) for the new Actions should be added to oozie-site.xml, under the property 'oozie.service.WorkflowSchemaService.ext.schemas'. A comma separated list for multiple Action schemas.</p> +<p>The executor class should be placed along with the oozie webapp in the correct path. Once Oozie is restarted, the custom action node can be used in workflows.</p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.0.0/DG_DistCpActionExtension.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/DG_DistCpActionExtension.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/DG_DistCpActionExtension.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/DG_DistCpActionExtension.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,359 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<hr /> +<a name="Oozie_DistCp_Action_Extension"></a> +<div class="section"><h2> Oozie DistCp Action Extension</h2> +<p><ul><ul><li><a href="#DistCp_Action">DistCp Action</a> +</li> +<li><a href="#Appendix_DistCp_XML-Schema">Appendix, DistCp XML-Schema</a> +<ul><li><a href="#AE.A_Appendix_A_DistCp_XML-Schema">AE.A Appendix A, DistCp XML-Schema</a> +<ul><li><a href="#DistCp_Action_Schema_Version_1.0">DistCp Action Schema Version 1.0</a> +</li> +<li><a href="#DistCp_Action_Schema_Version_0.2">DistCp Action Schema Version 0.2</a> +</li> +<li><a href="#DistCp_Action_Schema_Version_0.1">DistCp Action Schema Version 0.1</a> +</li> +</ul> +</li> +</ul> +</li> +</ul> +</ul> +</p> +<a name="DistCp_Action"></a> +<div class="section"><h3>DistCp Action</h3> +<p>The <tt>DistCp</tt> + action uses Hadoop distributed copy to copy files from one cluster to another or within the same cluster.</p> +<p><b>IMPORTANT:</b> + The DistCp action may not work properly with all configurations (secure, insecure) in all versions +of Hadoop. For example, distcp between two secure clusters is tested and works well. Same is true with two insecure +clusters. In cases where a secure and insecure clusters are involved, distcp will not work.</p> +<p>Both Hadoop clusters have to be configured with proxyuser for the Oozie process as explained +<a href="./DG_QuickStart.html#HadoopProxyUser">here</a> + on the Quick Start page.</p> +<p><b>Syntax:</b> +</p> +<p><pre> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="distcp-example"> + <distcp xmlns="uri:oozie:distcp-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> + <name-node>${nameNode1}</name-node> + <arg>${nameNode1}/path/to/input.txt</arg> + <arg>${nameNode2}/path/to/output.txt</arg> + </distcp> + <ok to="end"/> + <error to="fail"/> + </action> + ... +</workflow-app> +</pre></p> +<p>The first <tt>arg</tt> + indicates the input and the second <tt>arg</tt> + indicates the output. In the above example, the input is on <tt>namenode1</tt> + +and the output is on <tt>namenode2</tt> +.</p> +<p><b>IMPORTANT:</b> + If using the DistCp action between 2 secure clusters, the following property must be added to the <tt>configuration</tt> + of +the action: +<pre> +<property> + <name>oozie.launcher.mapreduce.job.hdfs-servers</name> + <value>${nameNode1},${nameNode2}</value> +</property> +</pre></p> +<p>The <tt>DistCp</tt> + action is also commonly used to copy files within the same cluster. Cases where copying files within +a directory to another directory or directories to target directory is supported. Example below will illustrate a +copy within a cluster, notice the source and target <tt>nameNode</tt> + is the same and use of =*= syntax is supported to +represent only child files or directories within a source directory. For the sake of the example, <tt>jobTracker</tt> + and <tt>resourceManager</tt> + +are synonymous.</p> +<p><b>Syntax:</b> +</p> +<p><pre> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="copy-example"> + <distcp xmlns="uri:oozie:distcp-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> + <name-node>${nameNode}</name-node> + <arg>${nameNode}/path/to/source/*</arg> + <arg>${nameNode}/path/to/target/</arg> + </distcp> + <ok to="end"/> + <error to="fail"/> + </action> + ... +</workflow-app> +</pre></p> +<a name="Appendix_DistCp_XML-Schema"></a> +</div> +<div class="section"><h3>Appendix, DistCp XML-Schema</h3> +<a name="AE.A_Appendix_A_DistCp_XML-Schema"></a> +<div class="section"><h4>AE.A Appendix A, DistCp XML-Schema</h4> +<a name="DistCp_Action_Schema_Version_1.0"></a> +<div class="section"><h5>DistCp Action Schema Version 1.0</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:distcp="uri:oozie:distcp-action:1.0" elementFormDefault="qualified" + targetNamespace="uri:oozie:distcp-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="distcp" type="distcp:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="distcp:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="DistCp_Action_Schema_Version_0.2"></a> +</div> +<div class="section"><h5>DistCp Action Schema Version 0.2</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:distcp="uri:oozie:distcp-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:distcp-action:0.2"> +. + <xs:element name="distcp" type="distcp:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="distcp:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="distcp:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<a name="DistCp_Action_Schema_Version_0.1"></a> +</div> +<div class="section"><h5>DistCp Action Schema Version 0.1</h5> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:distcp="uri:oozie:distcp-action:0.1" elementFormDefault="qualified" + targetNamespace="uri:oozie:distcp-action:0.1"> +. + <xs:element name="distcp" type="distcp:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="distcp:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="distcp:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> +</div> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.0.0/DG_EmailActionExtension.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/DG_EmailActionExtension.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/DG_EmailActionExtension.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/DG_EmailActionExtension.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,250 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<hr /> +<a name="Oozie_Email_Action_Extension"></a> +<div class="section"><h2> Oozie Email Action Extension</h2> +<p><ul><ul><li><a href="#a3.2.4_Email_action">3.2.4 Email action</a> +</li> +<li><a href="#AE.A_Appendix_A_Email_XML-Schema">AE.A Appendix A, Email XML-Schema</a> +</li> +</ul> +</ul> +</p> +<p><a name="EmailAction"></a> +</p> +<a name="a3.2.4_Email_action"></a> +<div class="section"><h5>3.2.4 Email action</h5> +<p>The <tt>email</tt> + action allows sending emails in Oozie from a workflow application. An email action must provide <tt>to</tt> + +addresses, <tt>cc</tt> + addresses (optional), <tt>bcc</tt> + addresses (optional), a <tt>subject</tt> + and a <tt>body</tt> +. +Multiple recipients of an email can be provided as comma separated addresses.</p> +<p>The email action is executed synchronously, and the workflow job will wait until the specified +emails are sent before continuing to the next action.</p> +<p>All values specified in the <tt>email</tt> + action can be parameterized (templatized) using EL expressions.</p> +<p><b>Syntax:</b> +</p> +<p><pre> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> + ... + <action name="[NODE-NAME]"> + <email xmlns="uri:oozie:email-action:0.2"> + <to>[COMMA-SEPARATED-TO-ADDRESSES]</to> + <cc>[COMMA-SEPARATED-CC-ADDRESSES]</cc> <!-- cc is optional --> + <bcc>[COMMA-SEPARATED-BCC-ADDRESSES]</bcc> <!-- bcc is optional --> + <subject>[SUBJECT]</subject> + <body>[BODY]</body> + <content_type>[CONTENT-TYPE]</content_type> <!-- content_type is optional --> + <attachment>[COMMA-SEPARATED-HDFS-FILE-PATHS]</attachment> <!-- attachment is optional --> + </email> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></p> +<p>The <tt>to</tt> + and <tt>cc</tt> + and <tt>bcc</tt> + commands are used to specify recipients who should get the mail. Multiple email recipients +can be provided using comma-separated values. Providing a <tt>to</tt> + command is necessary, while the <tt>cc</tt> + or <tt>bcc</tt> + may +optionally be used along.</p> +<p>The <tt>subject</tt> + and <tt>body</tt> + commands are used to specify subject and body of the mail. +From uri:oozie:email-action:0.2 one can also specify mail content type as <content_type>text/html</content_type>. +"text/plain" is default.</p> +<p>The <tt>attachment</tt> + is used to attach a file(s) on HDFS to the mail. Multiple attachment can be provided using comma-separated values. Non fully qualified path is considered as a file on default HDFS. A local file cannot be attached.</p> +<p><b>Configuration</b> +</p> +<p>The <tt>email</tt> + action requires some SMTP server configuration to be present (in oozie-site.xml). The following are the values +it looks for:</p> +<p><tt>oozie.email.smtp.host</tt> + - The host where the email action may find the SMTP server (localhost by default). +=oozie.email.smtp.port= - The port to connect to for the SMTP server (25 by default). +=oozie.email.from.address= - The from address to be used for mailing all emails (oozie@localhost by default). +=oozie.email.smtp.auth= - Boolean property that toggles if authentication is to be done or not. (false by default). +=oozie.email.smtp.username= - If authentication is enabled, the username to login as (empty by default). +=oozie.email.smtp.password= - If authentication is enabled, the username's password (empty by default). +=oozie.email.attachment.enabled= - Boolean property that toggles if configured attachments are to be placed into the emails. (false by default). +=oozie.email.smtp.socket.timeout.ms= - The timeout to apply over all SMTP server socket operations (10000ms by default).</p> +<p><b>Example:</b> +</p> +<p><pre> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> + ... + <action name="an-email"> + <email xmlns="uri:oozie:email-action:0.1"> + <to>b...@initech.com,the.other....@initech.com</to> + <cc>w...@initech.com</cc> + <bcc>yet.another....@initech.com</bcc> + <subject>Email notifications for ${wf:id()}</subject> + <body>The wf ${wf:id()} successfully completed.</body> + </email> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></p> +<p>In the above example, an email is sent to 'bob', 'the.other.bob', 'will' (cc), yet.another.bob (bcc) +with the subject and body both containing the workflow ID after substitution.</p> +<a name="AE.A_Appendix_A_Email_XML-Schema"></a> +</div> +<div class="section"><h4>AE.A Appendix A, Email XML-Schema</h4> +<p><pre> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:email="uri:oozie:email-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:email-action:0.2"> +. + <xs:element name="email" type="email:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="to" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="cc" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="bcc" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="subject" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="body" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="content_type" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="attachment" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +</xs:schema> +</pre></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.0.0/DG_Examples.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/DG_Examples.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/DG_Examples.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/DG_Examples.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,330 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="Oozie_Examples"></a> +<div class="section"><h2> Oozie Examples</h2> +<p><ul><ul><li><a href="#Command_Line_Examples">Command Line Examples</a> +<ul><li><a href="#Setting_Up_the_Examples">Setting Up the Examples</a> +</li> +<li><a href="#Running_the_Examples">Running the Examples</a> +</li> +</ul> +</li> +<li><a href="#Java_API_Example">Java API Example</a> +</li> +<li><a href="#Local_Oozie_Example">Local Oozie Example</a> +</li> +</ul> +</ul> +</p> +<a name="Command_Line_Examples"></a> +<div class="section"><h3>Command Line Examples</h3> +<a name="Setting_Up_the_Examples"></a> +<div class="section"><h4>Setting Up the Examples</h4> +<p>Oozie examples are bundled within the Oozie distribution in the <tt>oozie-examples.tar.gz</tt> + file.</p> +<p>Expanding this file will create an <tt>examples/</tt> + directory in the local file system.</p> +<p>The <tt>examples/</tt> + directory must be copied to the user HOME directory in HDFS:</p> +<p><pre> +$ hadoop fs -put examples examples +</pre></p> +<p><b>NOTE:</b> + If an examples directory already exists in HDFS, it must be deleted before copying it again. Otherwise files may not be +copied.</p> +<a name="Running_the_Examples"></a> +</div> +<div class="section"><h4>Running the Examples</h4> +<p>For the Streaming and Pig example, the <a href="./DG_QuickStart.html#OozieShareLib">Oozie Share Library</a> + must be installed in HDFS.</p> +<p>Add Oozie <tt>bin/</tt> + to the environment PATH.</p> +<p>The examples assume the ResourceManager is <tt>localhost:8032</tt> + and the NameNode is <tt><a href="./hdfs://localhost:8020.html">hdfs://localhost:8020</a> +</tt> +. If the actual +values are different, the job properties files in the examples directory must be edited to the correct values.</p> +<p>The example applications are under the examples/app directory, one directory per example. The directory contains the +application XML file (workflow, or workflow and coordinator), the <tt>job.properties</tt> + file to submit the job and any JAR +files the example may need.</p> +<p>The inputs for all examples are in the <tt>examples/input-data/</tt> + directory.</p> +<p>The examples create output under the <tt>examples/output-data/${EXAMPLE_NAME}</tt> + directory.</p> +<p><b>Note</b> +: The <tt>job.properties</tt> + file needs to be a local file during submissions, and not a HDFS path.</p> +<p><b>How to run an example application:</b> +</p> +<p><pre> +$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run +. +job: 14-20090525161321-oozie-tucu +</pre></p> +<p>Check the workflow job status:</p> +<p><pre> +$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-tucu +. +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +Workflow Name : map-reduce-wf +App Path : hdfs://localhost:8020/user/tucu/examples/apps/map-reduce +Status : SUCCEEDED +Run : 0 +User : tucu +Group : users +Created : 2009-05-26 05:01 +0000 +Started : 2009-05-26 05:01 +0000 +Ended : 2009-05-26 05:01 +0000 +Actions +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +Action Name Type Status Transition External Id External Status Error Code Start Time End Time +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +mr-node map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000 +.---------------------------------------------------------------------------------------------------------------------------------------------------------------- +</pre></p> +<p>To check the workflow job status via the Oozie web console, with a browser go to <tt><a class="externalLink" href="http://localhost:11000/oozie";>http://localhost:11000/oozie</a> +</tt> +.</p> +<p>To avoid having to provide the <tt>-oozie</tt> + option with the Oozie URL with every <tt>oozie</tt> + command, set <tt>OOZIE_URL</tt> + env +variable to the Oozie URL in the shell environment. For example:</p> +<p><pre> +$ export OOZIE_URL="http://localhost:11000/oozie" +$ +$ oozie job -info 14-20090525161321-oozie-tucu +</pre></p> +<a name="Java_API_Example"></a> +</div> +</div> +<div class="section"><h3>Java API Example</h3> +<p>Oozie provides a =<a href="./apidocs/org/org/apache/oozie/client/package-summary.html">Java Client API</a> + that simplifies +integrating Oozie with Java applications. This Java Client API is a convenience API to interact with Oozie Web-Services +API.</p> +<p>The following code snippet shows how to submit an Oozie job using the Java Client API.</p> +<p><pre> +import org.apache.oozie.client.OozieClient; +import org.apache.oozie.client.WorkflowJob; +. +import java.util.Properties; +. + ... +. + // get a OozieClient for local Oozie + OozieClient wc = new OozieClient("http://bar:11000/oozie"); +. + // create a workflow job configuration and set the workflow application path + Properties conf = wc.createConfiguration(); + conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app"); +. + // setting workflow parameters + conf.setProperty("resourceManager", "foo:8032"); + conf.setProperty("inputDir", "/usr/tucu/inputdir"); + conf.setProperty("outputDir", "/usr/tucu/outputdir"); + ... +. + // submit and start the workflow job + String jobId = wc.run(conf); + System.out.println("Workflow job submitted"); +. + // wait until the workflow job finishes printing the status every 10 seconds + while (wc.getJobInfo(jobId).getStatus() == Workflow.Status.RUNNING) { + System.out.println("Workflow job running ..."); + Thread.sleep(10 * 1000); + } +. + // print the final status of the workflow job + System.out.println("Workflow job completed ..."); + System.out.println(wf.getJobInfo(jobId)); + ... +</pre></p> +<a name="Local_Oozie_Example"></a> +</div> +<div class="section"><h3>Local Oozie Example</h3> +<p>Oozie provides an embedded Oozie implementation, <tt><a href="./apidocs/org/apache/oozie/local/LocalOozie.html">LocalOozie</a> +</tt> +, +which is useful for development, debugging and testing of workflow applications within the convenience of an IDE.</p> +<p>The code snippet below shows the usage of the <tt>LocalOozie</tt> + class. All the interaction with Oozie is done using Oozie + <tt>OozieClient</tt> + Java API, as shown in the previous section.</p> +<p>The examples bundled with Oozie include the complete and running class, <tt>LocalOozieExample</tt> + from where this snippet was +taken.</p> +<p><pre> +import org.apache.oozie.local.LocalOozie; +import org.apache.oozie.client.OozieClient; +import org.apache.oozie.client.WorkflowJob; +. +import java.util.Properties; +. + ... + // start local Oozie + LocalOozie.start(); +. + // get a OozieClient for local Oozie + OozieClient wc = LocalOozie.getClient(); +. + // create a workflow job configuration and set the workflow application path + Properties conf = wc.createConfiguration(); + conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app"); +. + // setting workflow parameters + conf.setProperty("resourceManager", "foo:8032"); + conf.setProperty("inputDir", "/usr/tucu/inputdir"); + conf.setProperty("outputDir", "/usr/tucu/outputdir"); + ... +. + // submit and start the workflow job + String jobId = wc.run(conf); + System.out.println("Workflow job submitted"); +. + // wait until the workflow job finishes printing the status every 10 seconds + while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) { + System.out.println("Workflow job running ..."); + Thread.sleep(10 * 1000); + } +. + // print the final status of the workflow job + System.out.println("Workflow job completed ..."); + System.out.println(wc.getJobInfo(jobId)); +. + // stop local Oozie + LocalOozie.stop(); + ... +</pre></p> +<p>Also asynchronous actions like FS action can be used / tested using <tt>LocalOozie</tt> + / <tt>OozieClient</tt> + API. Please see the module +=oozie-mini= for details like <tt>fs-decision.xml</tt> + workflow example.</p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<p></p> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.0.0/DG_HCatalogIntegration.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.0.0/DG_HCatalogIntegration.html?rev=1828722&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.0.0/DG_HCatalogIntegration.html (added) +++ oozie/site/trunk/content/resources/docs/5.0.0/DG_HCatalogIntegration.html Mon Apr 9 14:12:36 2018 @@ -0,0 +1,292 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at Apr 9, 2018 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie - </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.0.0"> + 5.0.0</a> + <span class="divider">/</span> + </li> + <li class="active ">Oozie - </li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-04-09</li> + <li id="projectVersion" class="pull-right"> + Version: 5.0.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p></p> +<p><a href="./index.html">::Go back to Oozie Documentation Index::</a> +</p> +<a name="HCatalog_Integration_Since_Oozie_4.x"></a> +<div class="section"><h2> HCatalog Integration (Since Oozie 4.x)</h2> +<p><ul><ul><li><a href="#HCatalog_Overview">HCatalog Overview</a> +<ul><li><a href="#HCatalog_notifications">HCatalog notifications</a> +</li> +</ul> +</li> +<li><a href="#Oozie_HCatalog_Integration">Oozie HCatalog Integration</a> +<ul><li><a href="#Oozie_Server_Configuration">Oozie Server Configuration</a> +</li> +<li><a href="#HCatalog_URI_Format">HCatalog URI Format</a> +</li> +<li><a href="#HCatalog_Libraries">HCatalog Libraries</a> +</li> +<li><a href="#Coordinator">Coordinator</a> +</li> +<li><a href="#Workflow">Workflow</a> +</li> +<li><a href="#Known_Issues">Known Issues</a> +</li> +</ul> +</li> +</ul> +</ul> +</p> +<a name="HCatalog_Overview"></a> +<div class="section"><h3>HCatalog Overview</h3> +<p>HCatalog is a table and storage management layer for Hadoop that enables users with different data processing +tools - Pig, MapReduce, and Hive - to more easily read and write data on the grid. HCatalog's table abstraction presents +users with a relational view of data in the Hadoop distributed file system (HDFS).</p> +<p>Read <a class="externalLink" href="http://incubator.apache.org/hcatalog/docs/r0.5.0/index.html";>HCatalog Documentation</a> + to know more about HCatalog. +Working with HCatalog using pig is detailed in +<a class="externalLink" href="http://incubator.apache.org/hcatalog/docs/r0.5.0/loadstore.html";>HCatLoader and HCatStorer</a> +. +Working with HCatalog using MapReduce directly is detailed in +<a class="externalLink" href="http://incubator.apache.org/hcatalog/docs/r0.5.0/inputoutput.html";>HCatInputFormat and HCatOutputFormat</a> +.</p> +<a name="HCatalog_notifications"></a> +<div class="section"><h4>HCatalog notifications</h4> +<p>HCatalog provides notifications through a JMS provider like ActiveMQ when a new partition is added to a table in the +database. This allows applications to consume those events and schedule the work that depends on them. In case of Oozie, +the notifications are used to determine the availability of HCatalog partitions defined as data dependencies in the +Coordinator and trigger workflows.</p> +<p>Read <a class="externalLink" href="http://incubator.apache.org/hcatalog/docs/r0.5.0/notification.html";>HCatalog Notification</a> + to know more about +notifications in HCatalog.</p> +<a name="Oozie_HCatalog_Integration"></a> +</div> +</div> +<div class="section"><h3>Oozie HCatalog Integration</h3> +<p>Oozie's Coordinators so far have been supporting HDFS directories as a input data dependency. When a HDFS URI +template is specified as a dataset and input events are defined in Coordinator for the dataset, Oozie performs data +availability checks by polling the HDFS directory URIs resolved based on the nominal time. When all the data +dependencies are met, the Coordinator's workflow is triggered which then consumes the available HDFS data.</p> +<p>With addition of HCatalog support, Coordinators also support specifying a set of HCatalog tables or table partitions as a dataset. +The workflow is triggered when the HCatalog table partitions are available and the workflow actions can then read the +partition data. A mix of HDFS and HCatalog dependencies can be specified as input data dependencies. +Similar to HDFS directories, HCatalog table partitions can also be specified as output dataset events.</p> +<p>With HDFS data dependencies, Oozie has to poll HDFS every time to determine the availability of a directory. +If the HCatalog server is configured to publish partition availability notifications to a JMS provider, Oozie can be +configured to subscribe to it and trigger jobs immediately. This pub-sub model reduces pressure on Namenode and also +cuts down on delays caused by polling intervals.</p> +<p>In the absence of a message bus in the deployment, Oozie will always +poll the HCatalog server directly for partition availability with the same frequency as the HDFS polling. Even when +subscribed to notifications, Oozie falls back to polling HCatalog server for partitions that were available before the +coordinator action was materialized and to deal with missed notifications due to system downtimes. The frequency of the +fallback polling is usually lower than the constant polling. Defaults are 10 minutes and 1 minute respectively.</p> +<a name="Oozie_Server_Configuration"></a> +<div class="section"><h4>Oozie Server Configuration</h4> +<p>Refer to <a href="./AG_Install.html#HCatalog_Configuration">HCatalog Configuration</a> + section of <a href="./AG_Install.html">Oozie Install</a> + +documentation for the Oozie server side configuration required to support HCatalog table partitions as a data dependency.</p> +<a name="HCatalog_URI_Format"></a> +</div> +<div class="section"><h4>HCatalog URI Format</h4> +<p>Oozie supports specifying HCatalog partitions as a data dependency through a URI notation. The HCatalog partition URI is +used to identify a set of table partitions: <a href="./hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US..html">hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US.</a> +</p> +<p>The format to specify a HCatalog table URI is:</p> +<p><a href="./hcat://[metastore.html">hcat://[metastore</a> + server]:[port]/[database name]/[table name]</p> +<p>The format to specify a HCatalog table partition URI is:</p> +<p><a href="./hcat://[metastore.html">hcat://[metastore</a> + server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...</p> +<p>For example, +<pre> + <dataset name="logs" frequency="${coord:days(1)}" + initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles"> + <uri-template> + hcat://myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA + </uri-template> + </dataset> +</pre></p> +<p>Post Oozie-4.3.0 release, Oozie also supports the multiple HCatalog servers in the URI. Each of the server needs to be +separated by single comma (,).</p> +<p>The format to specify a HCatalog table partition URI with multiple HCatalog server is:</p> +<p><a href="./hcat://[metastore_server]:[port],[metastore_server]:[port]/[database_name]/[table_name]/[partkey1]=[value];[partkey2]=[value];....html">hcat://[metastore_server]:[port],[metastore_server]:[port]/[database_name]/[table_name]/[partkey1]=[value];[partkey2]=[value];...</a> +</p> +<p>For example, +<pre> + <dataset name="logs" frequency="${coord:days(1)}" + initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles"> + <uri-template> + hcat://myhcatmetastore:9080,myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA + </uri-template> + </dataset> +</pre></p> +<p>The regex for parsing the multiple HCatalog URI is exposed via oozie-site.xml, So Users can modify if there is any +requirement. Key for the regex is: <tt>oozie.hcat.uri.regex.pattern</tt> +</p> +<p>For example, following has multiple HCatalog URI with multiple HCatalog servers. To understand this, Oozie will split them into +two HCatalog URIs. For splitting the URIs, above mentioned regex is used.</p> +<p><a href="./hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us,scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us.html">hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us,scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us</a> +</p> +<p>After split: (This is internal Oozie mechanism)</p> +<p><a href="./hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us.html">hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us</a> +</p> +<p><a href="./scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us.html">scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us</a> +</p> +<p><a name="HCatalogLibraries"></a> +</p> +<a name="HCatalog_Libraries"></a> +</div> +<div class="section"><h4>HCatalog Libraries</h4> +<p>A workflow action interacting with HCatalog requires the following jars in the classpath: +hcatalog-core.jar, hcatalog-pig-adapter.jar, webhcat-java-client.jar, hive-common.jar, hive-exec.jar, +hive-metastore.jar, hive-serde.jar and libfb303.jar. +hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath. The correct +version of HCatalog and hive jars should be placed in classpath based on the version of HCatalog installed on the cluster.</p> +<p>The jars can be added to the classpath of the action using one of the below ways.<ul><li>You can place the jars and hive-site.xml in the system shared library. The shared library for a pig, hive or java action can be overridden to include hcatalog shared libraries along with the action's shared library. Refer to <a href="./WorkflowFunctionalSpec.html.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3">Shared Libraries</a> + for more information. The oozie-sharelib-[version].tar.gz in the oozie distribution bundles the required HCatalog jars in a hcatalog sharelib. If using a different version of HCatalog than the one bundled in the sharelib, copy the required HCatalog jars from such version into the sharelib.</li> +<li>You can place the jars and hive-site.xml in the workflow application lib/ path.</li> +<li>You can specify the location of the jar files in <tt>archive</tt> + tag and the hive-site.xml in <tt>file</tt> + tag in the corresponding pig, hive or java action.</li> +</ul> +</p> +<a name="Coordinator"></a> +</div> +<div class="section"><h4>Coordinator</h4> +<p>Refer to <a href="./CoordinatorFunctionalSpec.html">Coordinator Functional Specification</a> + for more information about<ul><li>how to specify HCatalog partitions as a data dependency using input dataset events</li> +<li>how to specify HCatalog partitions as output dataset events</li> +<li>the various EL functions available to work with HCatalog dataset events and how to use them to access HCatalog partitions in pig, hive or java actions in a workflow.</li> +</ul> +</p> +<a name="Workflow"></a> +</div> +<div class="section"><h4>Workflow</h4> +<p>Refer to <a href="./WorkflowFunctionalSpec.html">Workflow Functional Specification</a> + for more information about<ul><li>how to drop HCatalog table/partitions in the prepare block of a action</li> +<li>the HCatalog EL functions available to use in workflows</li> +</ul> +</p> +<p>Refer to <a href="./DG_ActionAuthentication.html">Action Authentication</a> + for more information about<ul><li>how to access a secure HCatalog from any action (e.g. hive, pig, etc) in a workflow</li> +</ul> +</p> +<a name="Known_Issues"></a> +</div> +<div class="section"><h4>Known Issues</h4> +<p><ul><li>When rerunning a coordinator action without specifying -nocleanup option if the 'output-event' are hdfs directories, then they are deleted. But if the 'output-event' is a hcatalog partition, currently the partition is not dropped.</li> +</ul> +</p> +</div> +</div> + + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}