Added: oozie/site/trunk/content/resources/docs/5.2.1/DG_SparkActionExtension.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.1/DG_SparkActionExtension.html?rev=1886952&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.1/DG_SparkActionExtension.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.1/DG_SparkActionExtension.html Fri Feb 26 14:14:19 2021 @@ -0,0 +1,527 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2021-02-26 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20210226" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/"; class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.1"> + 5.2.1</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2021-02-26</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.1 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p><hr /> +<h1>Oozie Spark Action Extension</h1> +<ul> +<li><a href="#Spark_Action">Spark Action</a> +<ul> +<li><a href="#Spark_Action_Logging">Spark Action Logging</a></li> +<li><a href="#Spark_on_YARN">Spark on YARN</a></li> +<li><a href="#PySpark_with_Spark_Action">PySpark with Spark Action</a></li> +<li><a href="#Using_Symlink_in_jar">Using Symlink in <jar></a></li></ul></li> +<li><a href="#Appendix_Spark_XML-Schema">Appendix, Spark XML-Schema</a> +<ul> +<li><a href="#AE.A_Appendix_A_Spark_XML-Schema">AE.A Appendix A, Spark XML-Schema</a> +<ul> +<li><a href="#Spark_Action_Schema_Version_1.0">Spark Action Schema Version 1.0</a></li> +<li><a href="#Spark_Action_Schema_Version_0.2">Spark Action Schema Version 0.2</a></li> +<li><a href="#Spark_Action_Schema_Version_0.1">Spark Action Schema Version 0.1</a></li></ul></li></ul></li></ul> + +<div class="section"> +<h2><a name="Spark_Action"></a>Spark Action</h2> +<p>The <tt>spark</tt> action runs a Spark job.</p> +<p>The workflow job will wait until the Spark job completes before continuing to the next action.</p> +<p>To run the Spark job, you have to configure the <tt>spark</tt> action with the <tt>resource-manager</tt>, <tt>name-node</tt>, Spark <tt>master</tt> elements as well as the necessary elements, arguments and configuration.</p> +<p>Spark options can be specified in an element called <tt>spark-opts</tt>.</p> +<p>A <tt>spark</tt> action can be configured to create or delete HDFS directories before starting the Spark job.</p> +<p>Oozie EL expressions can be used in the inline configuration. Property values specified in the <tt>configuration</tt> element override values specified in the <tt>job-xml</tt> file.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <spark xmlns="uri:oozie:spark-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[SPARK SETTINGS FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <master>[SPARK MASTER URL]</master> + <mode>[SPARK MODE]</mode> + <name>[SPARK JOB NAME]</name> + <class>[SPARK MAIN CLASS]</class> + <jar>[SPARK DEPENDENCIES JAR / PYTHON FILE]</jar> + <spark-opts>[SPARK-OPTIONS]</spark-opts> + <arg>[ARG-VALUE]</arg> + ... + <arg>[ARG-VALUE]</arg> + ... + </spark> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>prepare</tt> element, if present, indicates a list of paths to delete or create before starting the job. Specified paths must start with <tt>hdfs://HOST:PORT</tt>.</p> +<p>The <tt>job-xml</tt> element, if present, specifies a file containing configuration for the Spark job. Multiple <tt>job-xml</tt> elements are allowed in order to specify multiple <tt>job.xml</tt> files.</p> +<p>The <tt>configuration</tt> element, if present, contains configuration properties that are passed to the Spark job.</p> +<p>The <tt>master</tt> element indicates the url of the Spark Master. Ex: <tt>spark://host:port</tt>, <tt>mesos://host:port</tt>, yarn-cluster, yarn-client, or local.</p> +<p>The <tt>mode</tt> element if present indicates the mode of spark, where to run spark driver program. Ex: client,cluster. This is typically not required because you can specify it as part of <tt>master</tt> (i.e. master<tt>yarn, mode</tt>client is equivalent to master=yarn-client). A local <tt>master</tt> always runs in client mode.</p> +<p>Depending on the <tt>master</tt> (and <tt>mode</tt>) entered, the Spark job will run differently as follows:</p> +<ul> + +<li>local mode: everything runs here in the Launcher Job.</li> +<li>yarn-client mode: the driver runs here in the Launcher Job and the executor in Yarn.</li> +<li>yarn-cluster mode: the driver and executor run in Yarn.</li> +</ul> +<p>The <tt>name</tt> element indicates the name of the spark application.</p> +<p>The <tt>class</tt> element if present, indicates the spark’s application main class.</p> +<p>The <tt>jar</tt> element indicates a comma separated list of jars or python files.</p> +<p>The <tt>spark-opts</tt> element, if present, contains a list of Spark options that can be passed to Spark. Spark configuration options can be passed by specifying ‘–conf key=value’ or other Spark CLI options. Values containing whitespaces can be enclosed by double quotes.</p> +<p>Some examples of the <tt>spark-opts</tt> element:</p> +<ul> + +<li>‘–conf key=value’</li> +<li>‘–conf key1=value1 value2’</li> +<li>‘–conf key1=“value1 value2”’</li> +<li>‘–conf key1=value1 key2=“value2 value3”’</li> +<li>‘–conf key=value –verbose –properties-file user.properties’</li> +</ul> +<p>There are several ways to define properties that will be passed to Spark. They are processed in the following order:</p> +<ul> + +<li>propagated from <tt>oozie.service.SparkConfigurationService.spark.configurations</tt></li> +<li>read from a localized <tt>spark-defaults.conf</tt> file</li> +<li>read from a file defined in <tt>spark-opts</tt> via the <tt>--properties-file</tt></li> +<li>properties defined in <tt>spark-opts</tt> element</li> +</ul> +<p>(The latter takes precedence over the former.) The server propagated properties, the <tt>spark-defaults.conf</tt> and the user-defined properties file are merged together into a single properties file as Spark handles only one file in its <tt>--properties-file</tt> option.</p> +<p>The <tt>arg</tt> element if present, contains arguments that can be passed to spark application.</p> +<p>In case some property values are present both in <tt>spark-defaults.conf</tt> and as property key/value pairs generated by Oozie, the user configured values from <tt>spark-defaults.conf</tt> are prepended to the ones generated by Oozie, as part of the Spark arguments list.</p> +<p>Following properties to prepend to Spark arguments:</p> +<ul> + +<li><tt>spark.executor.extraClassPath</tt></li> +<li><tt>spark.driver.extraClassPath</tt></li> +<li><tt>spark.executor.extraJavaOptions</tt></li> +<li><tt>spark.driver.extraJavaOptions</tt></li> +</ul> +<p>All the above elements can be parameterized (templatized) using EL expressions.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myfirstsparkjob"> + <spark xmlns="uri:oozie:spark-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <master>local[*]</master> + <mode>client</mode> + <name>Spark Example</name> + <class>org.apache.spark.examples.mllib.JavaALS</class> + <jar>/lib/spark-examples_2.10-1.1.0.jar</jar> + <spark-opts>--executor-memory 20G --num-executors 50 + --conf spark.executor.extraJavaOptions="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"</spark-opts> + <arg>inputpath=hdfs://localhost/input/file.txt</arg> + <arg>value=2</arg> + </spark> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<div class="section"> +<h3><a name="Spark_Action_Logging"></a>Spark Action Logging</h3> +<p>Spark action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Spark.</p> +<p>From Oozie web-console, from the Spark action pop up using the ‘Console URL’ link, it is possible to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.</p></div> +<div class="section"> +<h3><a name="Spark_on_YARN"></a>Spark on YARN</h3> +<p>To ensure that your Spark job shows up in the Spark History Server, make sure to specify these three Spark configuration properties either in <tt>spark-opts</tt> with <tt>--conf</tt> or from <tt>oozie.service.SparkConfigurationService.spark.configurations</tt> in oozie-site.xml.</p> +<ol style="list-style-type: decimal"> + +<li> + +<p>spark.yarn.historyServer.address=SPH-HOST:18088</p> +</li> +<li> + +<p>spark.eventLog.dir=<tt>hdfs://NN:8020/user/spark/applicationHistory</tt></p> +</li> +<li> + +<p>spark.eventLog.enabled=true</p> +</li> +</ol></div> +<div class="section"> +<h3><a name="PySpark_with_Spark_Action"></a>PySpark with Spark Action</h3> +<p>To submit PySpark scripts with Spark Action, pyspark dependencies must be available in sharelib or in workflow’s lib/ directory. For more information, please refer to <a href="AG_Install.html#Oozie_Share_Lib">installation document.</a></p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + .... + <action name="myfirstpysparkjob"> + <spark xmlns="uri:oozie:spark-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <master>yarn-cluster</master> + <name>Spark Example</name> + <jar>pi.py</jar> + <spark-opts>--executor-memory 20G --num-executors 50 + --conf spark.executor.extraJavaOptions="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"</spark-opts> + <arg>100</arg> + </spark> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>jar</tt> element indicates python file. Refer to the file by it’s localized name, because only local files are allowed in PySpark. The py file should be in the lib/ folder next to the workflow.xml or added using the <tt>file</tt> element so that it’s localized to the working directory with just its name.</p></div> +<div class="section"> +<h3><a name="Using_Symlink_in_jar"></a>Using Symlink in <jar></h3> +<p>A symlink must be specified using <a href="WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job">file</a> element. Then, you can use the symlink name in <tt>jar</tt> element.</p> +<p><b>Example:</b></p> +<p>Specifying relative path for symlink:</p> +<p>Make sure that the file is within the application directory i.e. <tt>oozie.wf.application.path</tt> .</p> + +<div> +<div> +<pre class="source"> <spark xmlns="uri:oozie:spark-action:1.0"> + ... + <jar>py-spark-example-symlink.py</jar> + ... + ... + <file>py-spark.py#py-spark-example-symlink.py</file> + ... + </spark> +</pre></div></div> + +<p>Specifying full path for symlink:</p> + +<div> +<div> +<pre class="source"> <spark xmlns="uri:oozie:spark-action:1.0"> + ... + <jar>spark-example-symlink.jar</jar> + ... + ... + <file>hdfs://localhost:8020/user/testjars/all-oozie-examples.jar#spark-example-symlink.jar</file> + ... + </spark> +</pre></div></div> +</div></div> +<div class="section"> +<h2><a name="Appendix_Spark_XML-Schema"></a>Appendix, Spark XML-Schema</h2> +<div class="section"> +<h3><a name="AE.A_Appendix_A_Spark_XML-Schema"></a>AE.A Appendix A, Spark XML-Schema</h3> +<div class="section"> +<h4><a name="Spark_Action_Schema_Version_1.0"></a>Spark Action Schema Version 1.0</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:spark="uri:oozie:spark-action:1.0" elementFormDefault="qualified" + targetNamespace="uri:oozie:spark-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="spark" type="spark:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="spark:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Spark_Action_Schema_Version_0.2"></a>Spark Action Schema Version 0.2</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:spark="uri:oozie:spark-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:spark-action:0.2"> + + <xs:element name="spark" type="spark:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="spark:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="spark:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Spark_Action_Schema_Version_0.1"></a>Spark Action Schema Version 0.1</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:spark="uri:oozie:spark-action:0.1" elementFormDefault="qualified" + targetNamespace="uri:oozie:spark-action:0.1"> + + <xs:element name="spark" type="spark:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="spark:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="spark:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +</pre></div></div> + +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2021 + <a href="https://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
Added: oozie/site/trunk/content/resources/docs/5.2.1/DG_SqoopActionExtension.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.1/DG_SqoopActionExtension.html?rev=1886952&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.1/DG_SqoopActionExtension.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.1/DG_SqoopActionExtension.html Fri Feb 26 14:14:19 2021 @@ -0,0 +1,432 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2021-02-26 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20210226" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/"; class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.1"> + 5.2.1</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2021-02-26</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.1 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p><hr /> +<h1>Oozie Sqoop Action Extension</h1> +<ul> +<li><a href="#Sqoop_Action">Sqoop Action</a> +<ul> +<li><a href="#Sqoop_Action_Counters">Sqoop Action Counters</a></li> +<li><a href="#Sqoop_Action_Logging">Sqoop Action Logging</a></li></ul></li> +<li><a href="#Appendix_Sqoop_XML-Schema">Appendix, Sqoop XML-Schema</a> +<ul> +<li><a href="#AE.A_Appendix_A_Sqoop_XML-Schema">AE.A Appendix A, Sqoop XML-Schema</a> +<ul> +<li><a href="#Sqoop_Action_Schema_Version_1.0">Sqoop Action Schema Version 1.0</a></li> +<li><a href="#Sqoop_Action_Schema_Version_0.3">Sqoop Action Schema Version 0.3</a></li> +<li><a href="#Sqoop_Action_Schema_Version_0.2">Sqoop Action Schema Version 0.2</a></li></ul></li></ul></li></ul> + +<div class="section"> +<h2><a name="Sqoop_Action"></a>Sqoop Action</h2> +<p><b>IMPORTANT:</b> The Sqoop action requires Apache Hadoop 1.x or 2.x.</p> +<p>The <tt>sqoop</tt> action runs a Sqoop job.</p> +<p>The workflow job will wait until the Sqoop job completes before continuing to the next action.</p> +<p>To run the Sqoop job, you have to configure the <tt>sqoop</tt> action with the <tt>resource-manager</tt>, <tt>name-node</tt> and Sqoop <tt>command</tt> or <tt>arg</tt> elements as well as configuration.</p> +<p>A <tt>sqoop</tt> action can be configured to create or delete HDFS directories before starting the Sqoop job.</p> +<p>Sqoop configuration can be specified with a file, using the <tt>job-xml</tt> element, and inline, using the <tt>configuration</tt> elements.</p> +<p>Oozie EL expressions can be used in the inline configuration. Property values specified in the <tt>configuration</tt> element override values specified in the <tt>job-xml</tt> file.</p> +<p>Note that YARN <tt>yarn.resourcemanager.address</tt> / <tt>resource-manager</tt> and HDFS <tt>fs.default.name</tt> / <tt>name-node</tt> properties must not be present in the inline configuration.</p> +<p>As with Hadoop <tt>map-reduce</tt> jobs, it is possible to add files and archives in order to make them available to the Sqoop job. Refer to the [WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job] section for more information about this feature.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <sqoop xmlns="uri:oozie:sqoop-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <command>[SQOOP-COMMAND]</command> + <arg>[SQOOP-ARGUMENT]</arg> + ... + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </sqoop> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>prepare</tt> element, if present, indicates a list of paths to delete or create before starting the job. Specified paths must start with <tt>hdfs://HOST:PORT</tt>.</p> +<p>The <tt>job-xml</tt> element, if present, specifies a file containing configuration for the Sqoop job. As of schema 0.3, multiple <tt>job-xml</tt> elements are allowed in order to specify multiple <tt>job.xml</tt> files.</p> +<p>The <tt>configuration</tt> element, if present, contains configuration properties that are passed to the Sqoop job.</p> +<p><b>Sqoop command</b></p> +<p>The Sqoop command can be specified either using the <tt>command</tt> element or multiple <tt>arg</tt> elements.</p> +<p>When using the <tt>command</tt> element, Oozie will split the command into multiple arguments. There are two command splitting algorithms in Oozie.</p> +<p>If <tt>oozie.action.sqoop.shellsplitter</tt> property is set to <tt>false</tt> Oozie will split the command on every space.</p> +<p>If <tt>oozie.action.sqoop.shellsplitter</tt> property is set to <tt>true</tt> Oozie will split the command like <tt>bash</tt> splits the commands. In this case it’s possible to group strings together using quotes. For instance oozie will split <tt>--query "select * from employee"</tt> into two tokens: <tt>--query</tt> and <tt>select * from employee</tt>.</p> +<p>The default value of the <tt>oozie.action.sqoop.shellsplitter</tt> property is <tt>false</tt>.</p> +<p>When using the <tt>arg</tt> elements, Oozie will pass each argument value as an argument to Sqoop.</p> +<p>Consult the Sqoop documentation for a complete list of valid Sqoop commands.</p> +<p>All the above elements can be parameterized (templatized) using EL expressions.</p> +<p><b>Examples:</b></p> +<p>Using the <tt>command</tt> element:</p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myfirsthivejob"> + <sqoop xmlns="uri:oozie:sqoop-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <command>import --connect jdbc:hsqldb:file:db.hsqldb --table TT --target-dir hdfs://localhost:8020/user/tucu/foo -m 1</command> + </sqoop> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The same Sqoop action using <tt>arg</tt> elements:</p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myfirstsqoopjob"> + <sqoop xmlns="uri:oozie:sqoop-action:1.0"> + <resource-manager>foo:8032</resource-manager> + <name-node>bar:8020</name-node> + <prepare> + <delete path="${jobOutput}"/> + </prepare> + <configuration> + <property> + <name>mapred.compress.map.output</name> + <value>true</value> + </property> + </configuration> + <arg>import</arg> + <arg>--connect</arg> + <arg>jdbc:hsqldb:file:db.hsqldb</arg> + <arg>--table</arg> + <arg>TT</arg> + <arg>--target-dir</arg> + <arg>hdfs://localhost:8020/user/tucu/foo</arg> + <arg>-m</arg> + <arg>1</arg> + </sqoop> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>NOTE: The <tt>arg</tt> elements syntax, while more verbose, allows to have spaces in a single argument, something useful when using free from queries.</p> +<div class="section"> +<h3><a name="Sqoop_Action_Counters"></a>Sqoop Action Counters</h3> +<p>The counters of the map-reduce job run by the Sqoop action are available to be used in the workflow via the <a href="WorkflowFunctionalSpec.html#HadoopCountersEL">hadoop:counters() EL function</a>.</p> +<p>If the Sqoop action run an import all command, the <tt>hadoop:counters()</tt> EL will return the aggregated counters of all map-reduce jobs run by the Sqoop import all command.</p></div> +<div class="section"> +<h3><a name="Sqoop_Action_Logging"></a>Sqoop Action Logging</h3> +<p>Sqoop action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Sqoop.</p> +<p>From Oozie web-console, from the Sqoop action pop up using the ‘Console URL’ link, it is possible to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.</p> +<p>The logging level of the Sqoop action can set in the Sqoop action configuration using the property <tt>oozie.sqoop.log.level</tt>. The default value is <tt>INFO</tt>.</p></div></div> +<div class="section"> +<h2><a name="Appendix_Sqoop_XML-Schema"></a>Appendix, Sqoop XML-Schema</h2> +<div class="section"> +<h3><a name="AE.A_Appendix_A_Sqoop_XML-Schema"></a>AE.A Appendix A, Sqoop XML-Schema</h3> +<div class="section"> +<h4><a name="Sqoop_Action_Schema_Version_1.0"></a>Sqoop Action Schema Version 1.0</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:sqoop="uri:oozie:sqoop-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:sqoop-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="sqoop" type="sqoop:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="sqoop:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice> + <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/> + </xs:choice> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Sqoop_Action_Schema_Version_0.3"></a>Sqoop Action Schema Version 0.3</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:sqoop="uri:oozie:sqoop-action:0.3" elementFormDefault="qualified" + targetNamespace="uri:oozie:sqoop-action:0.3"> + + <xs:element name="sqoop" type="sqoop:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice> + <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/> + </xs:choice> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="sqoop:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="sqoop:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Sqoop_Action_Schema_Version_0.2"></a>Sqoop Action Schema Version 0.2</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:sqoop="uri:oozie:sqoop-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:sqoop-action:0.2"> + + <xs:element name="sqoop" type="sqoop:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice> + <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/> + </xs:choice> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="sqoop:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="sqoop:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</pre></div></div> + +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2021 + <a href="https://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.2.1/DG_SshActionExtension.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.1/DG_SshActionExtension.html?rev=1886952&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.1/DG_SshActionExtension.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.1/DG_SshActionExtension.html Fri Feb 26 14:14:19 2021 @@ -0,0 +1,253 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2021-02-26 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20210226" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/"; class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.1"> + 5.2.1</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2021-02-26</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.1 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p><hr /> +<h1>Oozie Ssh Action Extension</h1> +<ul> +<li><a href="#Ssh_Action">Ssh Action</a></li> +<li><a href="#Appendix_Ssh_XML-Schema">Appendix, Ssh XML-Schema</a> +<ul> +<li><a href="#AE.A_Appendix_A_Ssh_XML-Schema">AE.A Appendix A, Ssh XML-Schema</a> +<ul> +<li><a href="#Ssh_Action_Schema_Version_0.2">Ssh Action Schema Version 0.2</a></li> +<li><a href="#Ssh_Action_Schema_Version_0.1">Ssh Action Schema Version 0.1</a></li></ul></li></ul></li></ul> + +<div class="section"> +<h2><a name="Ssh_Action"></a>Ssh Action</h2> +<p>The <tt>ssh</tt> action starts a shell command on a remote machine as a remote secure shell in background. The workflow job will wait until the remote shell command completes before continuing to the next action.</p> +<p>The shell command must be present in the remote machine and it must be available for execution via the command path.</p> +<p>The shell command is executed in the home directory of the specified user in the remote host.</p> +<p>The output (STDOUT) of the ssh job can be made available to the workflow job after the ssh job ends. This information could be used from within decision nodes. If the output of the ssh job is made available to the workflow job the shell command must follow the following requirements:</p> +<ul> + +<li>The format of the output must be a valid Java Properties file.</li> +<li>The size of the output must not exceed 2KB.</li> +</ul> +<p>Note: Ssh Action will fail if any output is written to standard error / output upon login (e.g. .bashrc of the remote user contains ls -a).</p> +<p>Note: Ssh Action will fail if oozie fails to ssh connect to host for action status check (e.g., the host is under heavy load, or network is bad) after a configurable number (3 by default) of retries. The first retry will wait a configurable period of time ( 3 seconds by default) before check. The following retries will wait 2 times of previous wait time.</p> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <ssh xmlns="uri:oozie:ssh-action:0.1"> + <host>[USER]@[HOST]</host> + <command>[SHELL]</command> + <args>[ARGUMENTS]</args> + ... + <capture-output/> + </ssh> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>host</tt> indicates the user and host where the shell will be executed.</p> +<p><b>IMPORTANT:</b> The <tt>oozie.action.ssh.allow.user.at.host</tt> property, in the <tt>oozie-site.xml</tt> configuration, indicates if an alternate user than the one submitting the job can be used for the ssh invocation. By default this property is set to <tt>true</tt>.</p> +<p>The <tt>command</tt> element indicates the shell command to execute.</p> +<p>The <tt>args</tt> element, if present, contains parameters to be passed to the shell command. If more than one <tt>args</tt> element is present they are concatenated in order. When an <tt>args</tt> element contains a space, even when quoted, it will be considered as separate arguments (i.e. “Hello World” becomes “Hello” and “World”). Starting with ssh schema 0.2, you can use the <tt>arg</tt> element (note that this is different than the <tt>args</tt> element) to specify arguments that have a space in them (i.e. “Hello World” is preserved as “Hello World”). You can use either <tt>args</tt> elements, <tt>arg</tt> elements, or neither; but not both in the same action.</p> +<p>If the <tt>capture-output</tt> element is present, it indicates Oozie to capture output of the STDOUT of the ssh command execution. The ssh command output must be in Java Properties file format and it must not exceed 2KB. From within the workflow definition, the output of an ssh action node is accessible via the =String action:output(String node, String key)= function (Refer to section ‘4.2.6 Action EL Functions’).</p> +<p>The configuration of the <tt>ssh</tt> action can be parameterized (templatized) using EL expressions.</p> +<p><b>Example:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="myssjob"> + <ssh xmlns="uri:oozie:ssh-action:0.1"> + <host>[email protected]<host> + <command>uploaddata</command> + <args>jdbc:derby://bar.com:1527/myDB</args> + <args>hdfs://foobar.com:8020/usr/tucu/myData</args> + </ssh> + <ok to="myotherjob"/> + <error to="errorcleanup"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>In the above example, the <tt>uploaddata</tt> shell command is executed with two arguments, <tt>jdbc:derby://foo.com:1527/myDB</tt> and <tt>hdfs://foobar.com:8020/usr/tucu/myData</tt>.</p> +<p>The <tt>uploaddata</tt> shell must be available in the remote host and available in the command path.</p> +<p>The output of the command will be ignored because the <tt>capture-output</tt> element is not present.</p></div> +<div class="section"> +<h2><a name="Appendix_Ssh_XML-Schema"></a>Appendix, Ssh XML-Schema</h2> +<div class="section"> +<h3><a name="AE.A_Appendix_A_Ssh_XML-Schema"></a>AE.A Appendix A, Ssh XML-Schema</h3> +<div class="section"> +<h4><a name="Ssh_Action_Schema_Version_0.2"></a>Ssh Action Schema Version 0.2</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:ssh="uri:oozie:ssh-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:ssh-action:0.2"> +. + <xs:element name="ssh" type="ssh:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="host" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:choice> + <xs:element name="args" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:choice> + <xs:element name="capture-output" type="ssh:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="FLAG"/> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Ssh_Action_Schema_Version_0.1"></a>Ssh Action Schema Version 0.1</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:ssh="uri:oozie:ssh-action:0.1" elementFormDefault="qualified" + targetNamespace="uri:oozie:ssh-action:0.1"> +. + <xs:element name="ssh" type="ssh:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="host" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="args" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="ssh:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="FLAG"/> +. +</xs:schema> +</pre></div></div> + +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2021 + <a href="https://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: oozie/site/trunk/content/resources/docs/5.2.1/DG_WorkflowReRun.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.1/DG_WorkflowReRun.html?rev=1886952&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.1/DG_WorkflowReRun.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.1/DG_WorkflowReRun.html Fri Feb 26 14:14:19 2021 @@ -0,0 +1,177 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2021-02-26 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20210226" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/"; class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.1"> + 5.2.1</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2021-02-26</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.1 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Workflow ReRun</h1> +<ul> +<li><a href="#Configs">Configs</a></li> +<li><a href="#Pre-Conditions">Pre-Conditions</a></li> +<li><a href="#ReRun">ReRun</a></li></ul> + +<div class="section"> +<h2><a name="Configs"></a>Configs</h2> +<ul> + +<li>oozie.wf.application.path</li> +<li>Only one of following two configurations is mandatory. Both should not be defined at the same time. +<ul> + +<li><tt>oozie.wf.rerun.skip.nodes</tt></li> +<li><tt>oozie.wf.rerun.failnodes</tt></li> +</ul> +</li> +<li>Skip nodes are comma separated list of action names. They can be any action nodes including decision node. If this value is empty, or null, and <tt>oozie.wf.rerun.failnodes</tt> is false, than the whole workflow is rerunned.</li> +<li>The valid value of <tt>oozie.wf.rerun.failnodes</tt> is true or false. By default its value is false. In case of setting this value to true, all the nodes where the status is OK will be skipped.</li> +<li>If secured hadoop version is used, the following two properties needs to be specified as well +<ul> + +<li>mapreduce.jobtracker.kerberos.principal</li> +<li>dfs.namenode.kerberos.principal.</li> +</ul> +</li> +<li>Configurations can be passed as -D param.</li> +</ul> + +<div> +<div> +<pre class="source">$ oozie job -oozie http://localhost:11000/oozie -rerun 14-20090525161321-oozie-joe -Doozie.wf.rerun.skip.nodes=<> +</pre></div></div> +</div> +<div class="section"> +<h2><a name="Pre-Conditions"></a>Pre-Conditions</h2> +<ul> + +<li>Workflow with id wfId should exist.</li> +<li>Workflow with id wfId should be in SUCCEEDED/KILLED/FAILED.</li> +<li>If specified , nodes in the config <tt>oozie.wf.rerun.skip.nodes</tt> must be completed successfully.</li> +</ul></div> +<div class="section"> +<h2><a name="ReRun"></a>ReRun</h2> +<ul> + +<li>Reloads the configs.</li> +<li>If no configuration is passed, existing coordinator/workflow configuration will be used. If configuration is passed then, it will be merged with existing workflow configuration. Input configuration will take the precedence.</li> +<li>Currently there is no way to remove an existing configuration but only override by passing a different value in the input configuration.</li> +<li>Creates a new Workflow Instance with the same wfId.</li> +<li>Deletes the actions that are not skipped from the DB and copies data from old Workflow Instance to new one for skipped actions.</li> +<li>Action handler will skip the nodes given in the config with the same exit transition as before.</li> +</ul> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2021 + <a href="https://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}