Added: oozie/site/trunk/content/resources/docs/5.1.0/DG_HiveActionExtension.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.1.0/DG_HiveActionExtension.html?rev=1849307&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.1.0/DG_HiveActionExtension.html (added) +++ oozie/site/trunk/content/resources/docs/5.1.0/DG_HiveActionExtension.html Wed Dec 19 15:42:08 2018 @@ -0,0 +1,570 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p><hr /> +<h1>Oozie Hive Action Extension</h1> +<ul> +<li><a href="#Hive_Action">Hive Action</a> +<ul> +<li><a href="#Hive_Default_and_Site_Configuration_Files">Hive Default and Site Configuration Files</a></li> +<li><a href="#Hive_Action_Logging">Hive Action Logging</a></li></ul></li> +<li><a href="#Appendix_Hive_XML-Schema">Appendix, Hive XML-Schema</a> +<ul> +<li><a href="#AE.A_Appendix_A_Hive_XML-Schema">AE.A Appendix A, Hive XML-Schema</a> +<ul> +<li><a href="#Hive_Action_Schema_Version_1.0">Hive Action Schema Version 1.0</a></li> +<li><a href="#Hive_Action_Schema_Version_0.6">Hive Action Schema Version 0.6</a></li> +<li><a href="#Hive_Action_Schema_Version_0.5">Hive Action Schema Version 0.5</a></li> +<li><a href="#Hive_Action_Schema_Version_0.4">Hive Action Schema Version 0.4</a></li> +<li><a href="#Hive_Action_Schema_Version_0.3">Hive Action Schema Version 0.3</a></li> +<li><a href="#Hive_Action_Schema_Version_0.2">Hive Action Schema Version 0.2</a></li></ul></li></ul></li></ul> + +<p><a name="HiveAction"></a></p> +<div class="section"> +<h2><a name="Hive_Action"></a>Hive Action</h2> +<p>The <tt>hive</tt> action runs a Hive job.</p> +<p>The workflow job will wait until the Hive job completes before continuing to the next action.</p> +<p>To run the Hive job, you have to configure the <tt>hive</tt> action with the <tt>resource-manager</tt>, <tt>name-node</tt> and Hive <tt>script</tt> (or Hive <tt>query</tt>) elements as well as the necessary parameters and configuration.</p> +<p>A <tt>hive</tt> action can be configured to create or delete HDFS directories before starting the Hive job.</p> +<p>Hive configuration can be specified with a file, using the <tt>job-xml</tt> element, and inline, using the <tt>configuration</tt> elements.</p> +<p>Oozie EL expressions can be used in the inline configuration. Property values specified in the <tt>configuration</tt> element override values specified in the <tt>job-xml</tt> file.</p> +<p>Note that YARN <tt>yarn.resourcemanager.address</tt> (<tt>resource-manager</tt>) and HDFS <tt>fs.default.name</tt> (<tt>name-node</tt>) properties must not be present in the inline configuration.</p> +<p>As with Hadoop <tt>map-reduce</tt> jobs, it is possible to add files and archives in order to make them available to the Hive job. Refer to the [WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job] section for more information about this feature.</p> +<p>Oozie Hive action supports Hive scripts with parameter variables, their syntax is <tt>${VARIABLES}</tt>.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[HIVE SETTINGS FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <script>[HIVE-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </hive> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>prepare</tt> element, if present, indicates a list of paths to delete or create before starting the job. Specified paths must start with <tt>hdfs://HOST:PORT</tt>.</p> +<p>The <tt>job-xml</tt> element, if present, specifies a file containing configuration for the Hive job. As of schema 0.3, multiple <tt>job-xml</tt> elements are allowed in order to specify multiple <tt>job.xml</tt> files.</p> +<p>The <tt>configuration</tt> element, if present, contains configuration properties that are passed to the Hive job.</p> +<p>The <tt>script</tt> element must contain the path of the Hive script to execute. The Hive script can be templatized with variables of the form <tt>${VARIABLE}</tt>. The values of these variables can then be specified using the <tt>params</tt> element.</p> +<p>The <tt>query</tt> element available from uri:oozie:hive-action:0.6, can be used instead of the <tt>script</tt> element. It allows for embedding queries within the <tt>worklfow.xml</tt> directly. Similar to the <tt>script</tt> element, it also allows for the templatization of variables in the form <tt>${VARIABLE}</tt>.</p> +<p>The <tt>params</tt> element, if present, contains parameters to be passed to the Hive script.</p> +<p>All the above elements can be parameterized (templatized) using EL expressions.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myfirsthivejob"> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <script>myscript.q</script> + <param>InputDir=/home/tucu/input-data</param> + <param>OutputDir=${jobOutput}</param> + </hive> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<div class="section"> +<h3><a name="Hive_Default_and_Site_Configuration_Files"></a>Hive Default and Site Configuration Files</h3> +<p>Hive (as of Hive 0.8) ignores a <tt>hive-default.xml</tt> file. As a result, Oozie (as of Oozie 3.4) ignores the <tt>oozie.hive.defaults</tt> property that was previously required by earlier versions of Oozie for the Hive action.</p></div> +<div class="section"> +<h3><a name="Hive_Action_Logging"></a>Hive Action Logging</h3> +<p>Hive action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Hive.</p> +<p>From Oozie web-console, from the Hive action pop up using the ‘Console URL’ link, it is possible to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.</p> +<p>The logging level of the Hive action can set in the Hive action configuration using the property <tt>oozie.hive.log.level</tt>. The default value is <tt>INFO</tt>.</p></div></div> +<div class="section"> +<h2><a name="Appendix_Hive_XML-Schema"></a>Appendix, Hive XML-Schema</h2> +<div class="section"> +<h3><a name="AE.A_Appendix_A_Hive_XML-Schema"></a>AE.A Appendix A, Hive XML-Schema</h3> +<div class="section"> +<h4><a name="Hive_Action_Schema_Version_1.0"></a>Hive Action Schema Version 1.0</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Hive_Action_Schema_Version_0.6"></a>Hive Action Schema Version 0.6</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.6" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.6"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Hive_Action_Schema_Version_0.5"></a>Hive Action Schema Version 0.5</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.5" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.5"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Hive_Action_Schema_Version_0.4"></a>Hive Action Schema Version 0.4</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.4" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.4"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Hive_Action_Schema_Version_0.3"></a>Hive Action Schema Version 0.3</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.3" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.3"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Hive_Action_Schema_Version_0.2"></a>Hive Action Schema Version 0.2</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:hive="uri:oozie:hive-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:0.2"> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></div></div> + +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
Added: oozie/site/trunk/content/resources/docs/5.1.0/DG_JMSNotifications.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.1.0/DG_JMSNotifications.html?rev=1849307&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.1.0/DG_JMSNotifications.html (added) +++ oozie/site/trunk/content/resources/docs/5.1.0/DG_JMSNotifications.html Wed Dec 19 15:42:08 2018 @@ -0,0 +1,333 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>JMS Notifications</h1> +<ul> +<li><a href="#Overview">Overview</a></li> +<li><a href="#Oozie_Server_Configuration">Oozie Server Configuration</a></li> +<li><a href="#Consuming_Notifications">Consuming Notifications</a> +<ul> +<li><a href="#Notification_types">Notification types</a></li> +<li><a href="#JMS_Topic">JMS Topic</a></li> +<li><a href="#JMS_Message_Format">JMS Message Format</a></li> +<li><a href="#JMS_Client">JMS Client</a> +<ul> +<li><a href="#Example">Example</a></li> +<li><a href="#Applying_Selectors">Applying Selectors</a></li></ul></li></ul></li></ul> + +<div class="section"> +<h2><a name="Overview"></a>Overview</h2> +<p>Since Oozie 4.0, Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss events. This provides an alternative to polling Oozie for Job or SLA related information and getting events as they happen without any delay. Clients can be written to consume these notifications and integrate with different monitoring and alerting systems.</p></div> +<div class="section"> +<h2><a name="Oozie_Server_Configuration"></a>Oozie Server Configuration</h2> +<p>Refer to <a href="AG_Install.html#Notifications_Configuration">Notifications Configuration</a> section of <a href="AG_Install.html">Oozie Install</a> documentation for the Oozie server side configuration required to support publishing notifications to a JMS Provider. The JNDI properties for the JMS provider, the topics to publish to and the notification types to publish (Job and/or SLA) need to be configured.</p></div> +<div class="section"> +<h2><a name="Consuming_Notifications"></a>Consuming Notifications</h2> +<div class="section"> +<h3><a name="Notification_types"></a>Notification types</h3> +<p>Job and SLA notifications are published to the configured JMS Provider on the configured topics.</p> +<p>Job status change notifications include job start, success, failure, suspended, etc. Currently only workflow job and coordinator action status change notifications are published.</p> +<p>SLA notifications include START_MET, END_MET, DURATION_MET, START_MISS, END_MISS, DURATION_MISS events and are published for a workflow job, workflow action or coordinator action for which SLA information is configured in the job xml. Refer to <a href="DG_SLAMonitoring.html#Configuring_SLA_in_Applications">SLA Configuration</a> for information on configuring SLA for a workflow or coordinator.</p></div> +<div class="section"> +<h3><a name="JMS_Topic"></a>JMS Topic</h3> +<p>Consumers interested in notification on events will require to know the JNDI properties to connect to the JMS provider. They will also need to know the JMS topic on which notifications for a particular job are published.</p> +<p>Oozie Client provides the following APIs :</p> + +<div> +<div> +<pre class="source">public JMSConnectionInfo getJMSConnectionInfo() +public String getJMSTopicName(String jobId) +</pre></div></div> + +<p>The JMSConnectionInfo exposes 3 methods:</p> + +<div> +<div> +<pre class="source">Properties getJNDIProperties(); +String getTopicPattern(AppType appType); +String getTopicPrefix(); +</pre></div></div> + +<p>The topic is obtained by concatenating topic prefix and the substituted value for topic pattern. The topic pattern can be a constant value like workflow or coordinator which the administrator has configured or a variable (either ${username} or ${jobId}). If ${username}, it has to be substituted with the name of the user who has submitted the job; and if ${jobId} it has to be substituted with the job Id. Administrators can chose to publish messages to topics containing user names to avoid having one topic containing all messages and all users having to apply selectors to filter the message they are interested in.</p> +<p>The getJMSTopicName API can be used if the job id is already known and will give the exact topic name to which the notifications for that job are published.</p></div> +<div class="section"> +<h3><a name="JMS_Message_Format"></a>JMS Message Format</h3> +<p>JMS messages published are <tt>javax.jms.TextMessage</tt>. The body contains JSON and the header contains multiple properties that can be used as selectors. The header properties are not repeated in the body of the message to keep the messages small.</p> +<p><b>Message Header:</b> <br /> The different header properties are:</p> +<ul> + +<li>msgType - Value can be JOB or SLA.</li> +<li>user - The user who submitted the job.</li> +<li>appName - Application name of the job.</li> +<li>appType - Type of the job. One of WORKFLOW_JOB, WORKFLOW_ACTION, COORDINATOR_ACTION</li> +<li>slaStatus - Applicable only to SLA messages. Value is one of NOT_STARTED, IN_PROCESS, MET, MISS</li> +<li>eventStatus - It takes one of the following values- START_MET, START_MISS, DURATION_MET, DURATION_MISS, END_MET, END_MISS for SLA notifications. It can take any of the following values- WAITING, STARTED, SUCCESS, SUSPEND, FAILURE for job notifications. Note that event status is different from the Job status. It is included in the header to provide better filtering. Below is the mapping of event status to the actual status of workflow job or coordinator action.</li> +</ul> +<p>WAITING = When a Coordinator Action is waiting for input dependency</p> +<p>STARTED = When the Workflow Job or Coordinator Action is in RUNNING state</p> +<p>SUCCESS = When the Workflow Job or Coordinator Action is in SUCCEEDED state</p> +<p>SUSPEND = When the Workflow Job or Coordinator Action is in SUSPENDED state</p> +<p>FAILURE = When the Workflow Job or Coordinator Action is in terminal state other than SUCCEEDED.</p> +<p><b>Message Body for Job Notifications:</b> <br /> Sample JSON response for different job and sla events as below.</p> + +<div> +<div> +<pre class="source">Workflow Job in RUNNING state: +{"status":"RUNNING","id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000} +</pre></div></div> + +<div> +<div> +<pre class="source">Workflow Job in FAILED state: +{"status":"FAILED","errorCode":"EL_ERROR","errorMessage":"variable [dummyvalue] cannot be resolved", +"id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000,"endTime":1366672183543} +</pre></div></div> + +<div> +<div> +<pre class="source">Workflow Job in SUCCEEDED state: +{"status":"SUCCEEDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C@1","endTime":1366676224154} +</pre></div></div> + +<div> +<div> +<pre class="source">Workflow Job in SUSPENDED state: +{"status":"SUSPENDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C@1"} +</pre></div></div> + +<div> +<div> +<pre class="source">Coordinator Action in WAITING state: +{"status":"WAITING","nominalTime":1310342400000,"missingDependency":"hdfs://gsbl90107.blue.com:8020/user/john/dir1/file1", +"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"} +</pre></div></div> + +<div> +<div> +<pre class="source">Coordinator Action in RUNNING state: +{"status":"RUNNING","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1", +"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"} +</pre></div></div> + +<div> +<div> +<pre class="source">Coordinator Action in SUCCEEDED state: +{"status":"SUCCEEDED","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1", +"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677082799} +</pre></div></div> + +<div> +<div> +<pre class="source">Coordinator Action in FAILED state: +{"status":"FAILED","errorCode":"E0101","errorMessage":"dummyError","nominalTime":1310342400000, +"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000, +"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677140818} +</pre></div></div> + +<p><b>Message Body for SLA Notifications:</b> <br /></p> + +<div> +<div> +<pre class="source">Workflow Job in sla END_MISS state: +{"id":"0000000-000000000000001-oozie-wrkf-C@1","parentId":"0000000-000000000000001-oozie-wrkf-C", +"expectedStartTime":1356998400000,"notificationMessage":"notification of start miss","actualStartTime":1357002000000, +"expectedDuration":-1, "actualDuration":3600,"expectedEndTime":1356998400000,"actualEndTime":1357002000000} +</pre></div></div> +</div> +<div class="section"> +<h3><a name="JMS_Client"></a>JMS Client</h3> +<p>Oozie provides a helper class JMSMessagingUtils for consumers to deserialize the JMS messages back to Java objects. The below method getEventMessage() expects a sub type of EventMessage. There are different implementations of EventMessage - WorkflowJobMessage, CoordinatorActionMessage and SLAMessage.</p> + +<div> +<div> +<pre class="source"><T extends EventMessage> T JMSMessagingUtils.getEventMessage(Message jmsMessage) +</pre></div></div> + +<div class="section"> +<h4><a name="Example"></a>Example</h4> +<p>Below is a sample code to consume notifications.</p> +<p>First, create the Oozie client and retrieve the JNDI properties to make a connection to the JMS server.</p> + +<div> +<div> +<pre class="source"> OozieClient oc = new OozieClient("http://localhost:11000/oozie"); + JMSConnectionInfo jmsInfo = oc.getJMSConnectionInfo(); + Properties jndiProperties = jmsInfo.getJNDIProperties(); + Context jndiContext = new InitialContext(jndiProperties); + String connectionFactoryName = (String) jndiContext.getEnvironment().get("connectionFactoryNames"); + ConnectionFactory connectionFactory = (ConnectionFactory) jndiContext.lookup(connectionFactoryName); + Connection connection = connectionFactory.createConnection(); + Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE); + String topicPrefix = jmsInfo.getTopicPrefix(); + String topicPattern = jmsInfo.getTopicPattern(AppType.WORKFLOW_JOB); + // Following code checks if the topic pattern is + //'username', then the topic name is set to the actual user submitting the job + String topicName = null; + if (topicPattern.equals("${username}")) { + topicName = "john"; + // Following code checks if the topic pattern is + //'jobId', then the topic name is set to the job id + } else if (topicPattern.equals("${jobId}")) { + topicName = "0000004-140328125200198-oozie-oozi-W"; + } + Destination topic = session.createTopic(topicPrefix+topicName); + MessageConsumer consumer = session.createConsumer(topic); + consumer.setMessageListener(this); + connection.start(); +</pre></div></div> + +<p>To start receiving messages, the JMS <a class="externalLink" href="http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html";>MessageListener</a> interface needs to be implemented. Also, its onMessage() method needs to be implemented. This method will be called whenever a message is available on the JMS bus.</p> + +<div> +<div> +<pre class="source"> public void onMessage(Message message) { + if (message.getStringProperty(JMSHeaderConstants.MESSAGE_TYPE).equals(MessageType.SLA.name())){ + SLAMessage slaMessage = JMSMessagingUtils.getEventMessage(message); + // Further processing + } + else if (message.getStringProperty(JMSHeaderConstants.APP_TYPE).equals(AppType.WORKFLOW_JOB.name())){ + WorkflowJobMessage wfJobMessage = JMSMessagingUtils.getEventMessage(message); + // Further processing + } + } +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Applying_Selectors"></a>Applying Selectors</h4> +<p>Below is a sample ActiveMQ text message header properties section.</p> + +<div> +<div> +<pre class="source">ActiveMQTextMessage +{properties = {appName = map-reduce-wf, msgType=JOB, appType=WORKFLOW_JOB, user=john, msgFormat=json, eventStatus=STARTED} ...} +</pre></div></div> + +<p>On the header properties, consumers can apply JMS selectors to filter messages from JMS provider. They are listed at <a href="../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html">JMSHeaderConstants</a></p> +<p>Sample use of selector to filter events related to Job which have failed and has a particular app-name</p> + +<div> +<div> +<pre class="source">String selector=JMSHeaderConstants.EVENT_STATUS + "='FAILURE' AND " + JMSHeaderConstants.APP_NAME + "='app-name'"; +MessageConsumer consumer = session.createConsumer(topic, selector); +</pre></div></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.1.0/DG_Overview.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.1.0/DG_Overview.html?rev=1849307&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.1.0/DG_Overview.html (added) +++ oozie/site/trunk/content/resources/docs/5.1.0/DG_Overview.html Wed Dec 19 15:42:08 2018 @@ -0,0 +1,178 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Oozie Workflow Overview</h1> +<p>Oozie is a server based <i>Workflow Engine</i> specialized in running workflow jobs with actions that run Hadoop Map/Reduce and Pig jobs.</p> +<p>Oozie is a Java Web-Application that runs in a Java servlet-container.</p> +<p>For the purposes of Oozie, a workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in a control dependency DAG (Directed Acyclic Graph). “control dependency” from one action to another means that the second action can’t run until the first action has completed.</p> +<p>Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to <a class="externalLink" href="http://www.jboss.org/jbossjbpm/";>JBOSS JBPM</a> jPDL).</p> +<p>Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.</p> +<p>Oozie uses a custom SecurityManager inside it’s launcher to catch exit() calls from the user code. Make sure to delegate checkExit() calls to Oozie’s SecurityManager if the user code uses its own SecurityManager. The Launcher also grants java.security.AllPermission by default to the user code.</p> +<p>Oozie workflows contain control flow nodes and action nodes.</p> +<p>Control flow nodes define the beginning and the end of a workflow ( <tt>start</tt>, <tt>end</tt> and <tt>fail</tt> nodes) and provide a mechanism to control the workflow execution path ( <tt>decision</tt>, <tt>fork</tt> and <tt>join</tt> nodes).</p> +<p>Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and Oozie sub-workflow. Oozie can be extended to support additional type of actions.</p> +<p>Oozie workflows can be parameterized (using variables like <tt>${inputDir}</tt> within the workflow definition). When submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different output directories) several identical workflow jobs can concurrently.</p> +<div class="section"> +<h2><a name="WordCount_Workflow_Example"></a>WordCount Workflow Example</h2> +<p><b>Workflow Diagram:</b></p> +<img src="./DG_Overview.png" alt="" /> + +<p><b>hPDL Workflow Definition:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name='wordcount-wf' xmlns="uri:oozie:workflow:0.1"> + <start to='wordcount'/> + <action name='wordcount'> + <map-reduce> + <job-tracker>${jobTracker}</job-tracker> + <name-node>${nameNode}</name-node> + <configuration> + <property> + <name>mapred.mapper.class</name> + <value>org.myorg.WordCount.Map</value> + </property> + <property> + <name>mapred.reducer.class</name> + <value>org.myorg.WordCount.Reduce</value> + </property> + <property> + <name>mapred.input.dir</name> + <value>${inputDir}</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>${outputDir}</value> + </property> + </configuration> + </map-reduce> + <ok to='end'/> + <error to='end'/> + </action> + <kill name='kill'> + <message>Something went wrong: ${wf:errorCode('wordcount')}</message> + </kill/> + <end name='end'/> +</workflow-app> +</pre></div></div> + +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.1.0/DG_Overview.png URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.1.0/DG_Overview.png?rev=1849307&view=auto ============================================================================== Binary file - no diff available. Propchange: oozie/site/trunk/content/resources/docs/5.1.0/DG_Overview.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: oozie/site/trunk/content/resources/docs/5.1.0/DG_QuickStart.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.1.0/DG_QuickStart.html?rev=1849307&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.1.0/DG_QuickStart.html (added) +++ oozie/site/trunk/content/resources/docs/5.1.0/DG_QuickStart.html Wed Dec 19 15:42:08 2018 @@ -0,0 +1,338 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Oozie Quick Start</h1> +<p>These instructions install and run Oozie using an embedded Jetty server and an embedded Derby database.</p> +<p>For detailed install and configuration instructions refer to <a href="AG_Install.html">Oozie Install</a>.</p> +<ul> +<li><a href="#Building_Oozie">Building Oozie</a> +<ul> +<li><a href="#System_Requirements:">System Requirements:</a></li> +<li><a href="#Building_Oozie">Building Oozie</a></li></ul></li> +<li><a href="#Server_Installation">Server Installation</a> +<ul> +<li><a href="#System_Requirements">System Requirements</a></li> +<li><a href="#Server_Installation">Server Installation</a></li></ul></li> +<li><a href="#Client_Installation">Client Installation</a> +<ul> +<li><a href="#System_Requirements">System Requirements</a></li> +<li><a href="#Client_Installation">Client Installation</a></li></ul></li> +<li><a href="#Oozie_Share_Lib_Installation">Oozie Share Lib Installation</a></li></ul> + +<div class="section"> +<h2><a name="Building_Oozie"></a>Building Oozie</h2> +<div class="section"> +<h3><a name="System_Requirements:"></a>System Requirements:</h3> +<ul> + +<li>Unix box (tested on Mac OS X and Linux)</li> +<li>Java JDK 1.8+</li> +<li>Maven 3.0.1+</li> +<li>Hadoop 2.6.0+</li> +<li>Pig 0.10.1+</li> +</ul> +<p>JDK commands (java, javac) must be in the command path.</p> +<p>The Maven command (mvn) must be in the command path.</p></div> +<div class="section"> +<h3><a name="Building_Oozie"></a>Building Oozie</h3> +<p>Download a source distribution of Oozie from the “Releases” drop down menu on the <a class="externalLink" href="http://oozie.apache.org";>Oozie site</a>.</p> +<p>Expand the source distribution <tt>tar.gz</tt> and change directories into it.</p> +<p>The simplest way to build Oozie is to run the <tt>mkdistro.sh</tt> script:</p> + +<div> +<div> +<pre class="source">$ bin/mkdistro.sh [-DskipTests] +</pre></div></div> + +<p>Running <tt>mkdistro.sh</tt> will create the binary distribution of Oozie. By default, oozie war will not contain hadoop and hcatalog libraries, however they are required for oozie to work. There are 2 options to add these libraries:</p> +<ol style="list-style-type: decimal"> + +<li> + +<p>At install time, copy the hadoop and hcatalog libraries to libext and run oozie-setup.sh to setup Oozie. This is suitable when same oozie package needs to be used in multiple set-ups with different hadoop/hcatalog versions.</p> +</li> +<li> + +<p>Build with -Puber which will bundle the required libraries in the oozie war. Further, the following options are available to customise the versions of the dependencies:</p> + +<div> +<div> +<pre class="source">-Dhadoop.version=<version> - default 2.6.0 +-Ptez - Bundle tez jars in hive and pig sharelibs. Useful if you want to use tez ++as the execution engine for those applications. +-Dpig.version=<version> - default 0.16.0 +-Dpig.classifier=<classifier> - default h2 +-Dsqoop.version=<version> - default 1.4.3 +-Dsqoop.classifier=<classifier> - default hadoop100 +-Djetty.version=<version> - default 9.3.20.v20170531 +-Dopenjpa.version=<version> - default 2.2.2 +-Dxerces.version=<version> - default 2.10.0 +-Dcurator.version=<version> - default 2.5.0 +-Dhive.version=<version - default 1.2.0 +-Dhbase.version=<version> - default 1.2.3 +-Dtez.version=<version> - default 0.8.4 + +*IMPORTANT:* Profile hadoop-3 must be activated if building against Hadoop 3 +</pre></div></div> + +<p>More details on building Oozie can be found on the <a href="ENG_Building.html">Building Oozie</a> page.</p> +</li> +</ol></div></div> +<div class="section"> +<h2><a name="Server_Installation"></a>Server Installation</h2> +<div class="section"> +<h3><a name="System_Requirements"></a>System Requirements</h3> +<ul> + +<li>Unix (tested in Linux and Mac OS X)</li> +<li>Java 1.8+</li> +<li>Hadoop +<ul> + +<li><a class="externalLink" href="http://hadoop.apache.org";>Apache Hadoop</a> (tested with 1.2.1 & 2.6.0+)</li> +</ul> +</li> +<li>ExtJS library (optional, to enable Oozie webconsole) +<ul> + +<li><a class="externalLink" href="http://archive.cloudera.com/gplextras/misc/ext-2.2.zip";>ExtJS 2.2</a></li> +</ul> +</li> +</ul> +<p>The Java 1.8+ <tt>bin</tt> directory should be in the command path.</p></div> +<div class="section"> +<h3><a name="Server_Installation"></a>Server Installation</h3> +<p><b>IMPORTANT:</b> Oozie ignores any set value for <tt>OOZIE_HOME</tt>, Oozie computes its home automatically.</p> +<ul> + +<li>Build an Oozie binary distribution</li> +<li>Download a Hadoop binary distribution</li> +<li>Download ExtJS library (it must be version 2.2)</li> +</ul> +<p><b>NOTE:</b> The ExtJS library is not bundled with Oozie because it uses a different license.</p> +<p><b>NOTE:</b> Oozie UI browser compatibility Chrome (all), Firefox (3.5), Internet Explorer (8.0), Opera (10.5).</p> +<p><b>NOTE:</b> It is recommended to use a Oozie Unix user for the Oozie server.</p> +<p>Expand the Oozie distribution <tt>tar.gz</tt>.</p> +<p>Expand the Hadoop distribution <tt>tar.gz</tt> (as the Oozie Unix user).</p> +<p><a name="HadoopProxyUser"></a></p> +<p><b>NOTE:</b> Configure the Hadoop cluster with proxyuser for the Oozie process.</p> +<p>The following two properties are required in Hadoop core-site.xml:</p> + +<div> +<div> +<pre class="source"> <!-- OOZIE --> + <property> + <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name> + <value>[OOZIE_SERVER_HOSTNAME]</value> + </property> + <property> + <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name> + <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value> + </property> +</pre></div></div> + +<p>Replace the capital letter sections with specific values and then restart Hadoop.</p> +<p>The ExtJS library is optional (only required for the Oozie web-console to work)</p> +<p><b>IMPORTANT:</b> all Oozie server scripts (<tt>oozie-setup.sh</tt>, <tt>oozied.sh</tt>, <tt>oozie-start.sh</tt>, <tt>oozie-run.sh</tt> and <tt>oozie-stop.sh</tt>) run only under the Unix user that owns the Oozie installation directory, if necessary use <tt>sudo -u OOZIE_USER</tt> when invoking the scripts.</p> +<p>As of Oozie 3.3.2, use of <tt>oozie-start.sh</tt>, <tt>oozie-run.sh</tt>, and <tt>oozie-stop.sh</tt> has been deprecated and will print a warning. The <tt>oozied.sh</tt> script should be used instead; passing it <tt>start</tt>, <tt>run</tt>, or <tt>stop</tt> as an argument will perform the behaviors of <tt>oozie-start.sh</tt>, <tt>oozie-run.sh</tt>, and <tt>oozie-stop.sh</tt> respectively.</p> +<p>Create a <b>libext/</b> directory in the directory where Oozie was expanded.</p> +<p>If using the ExtJS library copy the ZIP file to the <b>libext/</b> directory. If hadoop and hcatalog libraries are not already included in the war, add the corresponding libraries to <b>libext/</b> directory.</p> +<p>A “sharelib create -fs fs_default_name [-locallib sharelib]” command is available when running oozie-setup.sh for uploading new sharelib into hdfs where the first argument is the default fs name and the second argument is the Oozie sharelib to install, it can be a tarball or the expanded version of it. If the second argument is omitted, the Oozie sharelib tarball from the Oozie installation directory will be used. Upgrade command is deprecated, one should use create command to create new version of sharelib. Sharelib files are copied to new <tt>lib_<timestamped></tt> directory. At start, server picks the sharelib from latest time-stamp directory. While starting server also purge sharelib directory which is older than sharelib retention days (defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).</p> +<p>db create|upgrade|postupgrade -run [-sqlfile <FILE>] command is for create, upgrade or postupgrade oozie db with an optional sql file</p> +<p>Run the <tt>oozie-setup.sh</tt> script to configure Oozie with all the components added to the <b>libext/</b> directory.</p> + +<div> +<div> +<pre class="source">$ bin/oozie-setup.sh sharelib create -fs <FS_URI> [-locallib <PATH>] + sharelib upgrade -fs <FS_URI> [-locallib <PATH>] + db create|upgrade|postupgrade -run [-sqlfile <FILE>] +</pre></div></div> + +<p><b>IMPORTANT</b>: If the Oozie server needs to establish secure connection with an external server with a self-signed certificate, make sure you specify the location of a truststore that contains required certificates. It can be done by configuring <tt>oozie.https.truststore.file</tt> in <tt>oozie-site.xml</tt>, or by setting the <tt>javax.net.ssl.trustStore</tt> system property. If it is set in both places, the value passed as system property will be used.</p> +<p>Create the Oozie DB using the ‘ooziedb.sh’ command line tool:</p> + +<div> +<div> +<pre class="source">$ bin/ooziedb.sh create -sqlfile oozie.sql -run + +Validate DB Connection. +DONE +Check DB schema does not exist +DONE +Check OOZIE_SYS table does not exist +DONE +Create SQL schema +DONE +DONE +Create OOZIE_SYS table +DONE + +Oozie DB has been created for Oozie version '3.2.0' + +$ +</pre></div></div> + +<p>Start Oozie as a daemon process run:</p> + +<div> +<div> +<pre class="source">$ bin/oozied.sh start +</pre></div></div> + +<p>To start Oozie as a foreground process run:</p> + +<div> +<div> +<pre class="source">$ bin/oozied.sh run +</pre></div></div> + +<p>Check the Oozie log file <tt>logs/oozie.log</tt> to ensure Oozie started properly.</p> +<p>Using the Oozie command line tool check the status of Oozie:</p> + +<div> +<div> +<pre class="source">$ bin/oozie admin -oozie http://localhost:11000/oozie -status +</pre></div></div> + +<p>Using a browser go to the <a class="externalLink" href="http://localhost:11000/oozie.html";>Oozie web console</a>, Oozie status should be <b>NORMAL</b>.</p> +<p>Refer to the <a href="DG_Examples.html">Running the Examples</a> document for details on running the examples.</p></div></div> +<div class="section"> +<h2><a name="Client_Installation"></a>Client Installation</h2> +<div class="section"> +<h3><a name="System_Requirements"></a>System Requirements</h3> +<ul> + +<li>Unix (tested in Linux and Mac OS X)</li> +<li>Java 1.8+</li> +</ul> +<p>The Java 1.8+ <tt>bin</tt> directory should be in the command path.</p></div> +<div class="section"> +<h3><a name="Client_Installation"></a>Client Installation</h3> +<p>Copy and expand the <tt>oozie-client</tt> TAR.GZ file bundled with the distribution. Add the <tt>bin/</tt> directory to the <tt>PATH</tt>.</p> +<p>Refer to the <a href="DG_CommandLineTool.html">Command Line Interface Utilities</a> document for a full reference of the <tt>oozie</tt> command line tool.</p> +<p>NOTE: The Oozie server installation includes the Oozie client. The Oozie client should be installed in remote machines only.</p> +<p><a name="OozieShareLib"></a></p></div></div> +<div class="section"> +<h2><a name="Oozie_Share_Lib_Installation"></a>Oozie Share Lib Installation</h2> +<p>Oozie share lib has been installed by oozie-setup.sh create command explained in the earlier section.</p> +<p>See the <a href="WorkflowFunctionalSpec.html#ShareLib">Workflow Functional Specification</a> and <a href="AG_Install.html#Oozie_Share_Lib">Installation</a> for more information about the Oozie ShareLib.</p> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}
