Repository: oozie Updated Branches: refs/heads/master f492a43cf -> 50e6c35b4
OOZIE-2600 OYA: Update Documentation (andras.piros via gezapeti) Project: http://git-wip-us.apache.org/repos/asf/oozie/repo Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/50e6c35b Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/50e6c35b Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/50e6c35b Branch: refs/heads/master Commit: 50e6c35b4ebad33c74ea06cb70d5f6848a36530d Parents: f492a43 Author: Gezapeti Cseh <gezap...@apache.org> Authored: Wed Mar 21 12:19:01 2018 +0100 Committer: Gezapeti Cseh <gezap...@apache.org> Committed: Wed Mar 21 12:19:01 2018 +0100 ---------------------------------------------------------------------- docs/src/site/twiki/AG_Install.twiki | 78 +++ .../site/twiki/DG_DistCpActionExtension.twiki | 44 +- docs/src/site/twiki/DG_Examples.twiki | 9 +- .../site/twiki/DG_Hive2ActionExtension.twiki | 55 +- .../src/site/twiki/DG_HiveActionExtension.twiki | 55 +- .../site/twiki/DG_ShellActionExtension.twiki | 58 +- .../site/twiki/DG_SparkActionExtension.twiki | 84 ++- .../site/twiki/DG_SqoopActionExtension.twiki | 63 ++- docs/src/site/twiki/DG_SshActionExtension.twiki | 4 +- docs/src/site/twiki/ENG_MiniOozie.twiki | 6 +- .../src/site/twiki/WorkflowFunctionalSpec.twiki | 555 ++++++++++++++++--- release-log.txt | 1 + 12 files changed, 854 insertions(+), 158 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/AG_Install.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/AG_Install.twiki b/docs/src/site/twiki/AG_Install.twiki index 1504215..8e2d795 100644 --- a/docs/src/site/twiki/AG_Install.twiki +++ b/docs/src/site/twiki/AG_Install.twiki @@ -296,6 +296,84 @@ Oozie resolves configuration property values in the following order: *NOTE:* The =oozie-default.xml= file found in Oozie's =conf/= directory is not used by Oozie, it is there for reference purposes only. +---+++ Precedence of Configuration Properties + +For compatibility reasons across Hadoop / Oozie versions, some configuration properties can be defined using multiple keys +in the launcher configuration. Beginning with Oozie 5.0.0, some of them can be overridden, some others will be prepended to default +configuration values. + +---++++ Overriding Configuration Values + +Overriding happens for following configuration entries with =oozie.launcher= prefix, by switching =oozie.launcher.override= +(on by default). + +For those, following is the general approach: + * check whether a YARN compatible entry is present. If yes, use it to override default value + * check whether a MapReduce v2 compatible entry is present. If yes, use it to override default value + * check whether a MapReduce v1 compatible entry is present. If yes, use it to override default value + * use default value + +Such properties are (legend: YARN / MapReduce v2 / MapReduce v1): + * max attempts of the MapReduce Application Master: + * N / A + * =mapreduce.map.maxattempts= + * =mapred.map.max.attempts= + * memory amount in MB of the MapReduce Application Master: + * =yarn.app.mapreduce.am.resource.mb= + * =mapreduce.map.memory.mb= + * =mapred.job.map.memory.mb= + * CPU vcore count of the MapReduce Application Master: + * =yarn.app.mapreduce.am.resource.cpu-vcores= + * =mapreduce.map.cpu.vcores= + * N / A + * logging level of the MapReduce Application Master: + * N / A + * =mapreduce.map.log.level= + * =mapred.map.child.log.level= + * MapReduce Application Master JVM options: + * =yarn.app.mapreduce.am.command-opts= + * =mapreduce.map.java.opts= + * =mapred.child.java.opts= + * MapReduce Application Master environment variable settings: + * =yarn.app.mapreduce.am.env= + * =mapreduce.map.env= + * =mapred.child.env= + * MapReduce Application Master job priority: + * N / A + * =mapreduce.job.priority= + * =mapred.job.priority= + * MapReduce Application Master job queue name: + * N / A + * =mapreduce.job.queuename= + * =mapred.job.queue.name= + * MapReduce View ACL settings: + * N / A + * =mapreduce.job.acl-view-job= + * N / A + * MapReduce Modify ACL settings: + * N / A + * =mapreduce.job.acl-modify-job= + * N / A + +This list can be extended or modified by adding new configuration entries or updating existing values +beginning with =oozie.launcher.override.= within =oozie-site.xml=. Examples can be found in =oozie-default.xml=. + +---++++ Prepending Configuration Values + +Prepending happens for following configuration entries with =oozie.launcher= prefix, by switching =oozie.launcher.prepend= +(on by default). + +For those, following is the general approach: + * check whether a YARN compatible entry is present. If yes, use it to prepend to default value + * use default value + +Such properties are (legend: YARN only): + * MapReduce Application Master JVM options: =yarn.app.mapreduce.am.admin-command-opts= + * MapReduce Application Master environment settings: =yarn.app.mapreduce.am.admin.user.env= + +This list can be extended or modified by adding new configuration entries or updating existing values +beginning with =oozie.launcher.prepend.= within =oozie-site.xml=. Examples can be found in =oozie-default.xml=. + ---+++ Logging Configuration By default, Oozie log configuration is defined in the =oozie-log4j.properties= configuration file. http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_DistCpActionExtension.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_DistCpActionExtension.twiki b/docs/src/site/twiki/DG_DistCpActionExtension.twiki index 260cd25..8bab3da 100644 --- a/docs/src/site/twiki/DG_DistCpActionExtension.twiki +++ b/docs/src/site/twiki/DG_DistCpActionExtension.twiki @@ -22,11 +22,11 @@ Both Hadoop clusters have to be configured with proxyuser for the Oozie process *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.4"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="distcp-example"> - <distcp xmlns="uri:oozie:distcp-action:0.2"> - <job-tracker>${jobTracker}</job-tracker> + <distcp xmlns="uri:oozie:distcp-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode1}</name-node> <arg>${nameNode1}/path/to/input.txt</arg> <arg>${nameNode2}/path/to/output.txt</arg> @@ -53,17 +53,17 @@ the action: The =DistCp= action is also commonly used to copy files within the same cluster. Cases where copying files within a directory to another directory or directories to target directory is supported. Example below will illustrate a copy within a cluster, notice the source and target =nameNode= is the same and use of =*= syntax is supported to -represent only child files or directories within a source directory. For the sake of the example, =jobTracker= and -=resourceManager= are synonymous. +represent only child files or directories within a source directory. For the sake of the example, =jobTracker= and =resourceManager= +are synonymous. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.4"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="copy-example"> - <distcp xmlns="uri:oozie:distcp-action:0.2"> - <job-tracker>${resourceManager}</job-tracker> + <distcp xmlns="uri:oozie:distcp-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <arg>${nameNode}/path/to/source/*</arg> <arg>${nameNode}/path/to/target/</arg> @@ -79,6 +79,34 @@ represent only child files or directories within a source directory. For the sak ---+++ AE.A Appendix A, DistCp XML-Schema +---++++ DistCp Action Schema Version 1.0 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:distcp="uri:oozie:distcp-action:1.0" elementFormDefault="qualified" + targetNamespace="uri:oozie:distcp-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="distcp" type="distcp:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="distcp:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</verbatim> + ---++++ DistCp Action Schema Version 0.2 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_Examples.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_Examples.twiki b/docs/src/site/twiki/DG_Examples.twiki index 87cac48..13dfa28 100644 --- a/docs/src/site/twiki/DG_Examples.twiki +++ b/docs/src/site/twiki/DG_Examples.twiki @@ -29,7 +29,7 @@ For the Streaming and Pig example, the [[DG_QuickStart#OozieShareLib][Oozie Shar Add Oozie =bin/= to the environment PATH. -The examples assume the JobTracker is =localhost:8021= and the NameNode is =hdfs://localhost:8020=. If the actual +The examples assume the ResourceManager is =localhost:8032= and the NameNode is =hdfs://localhost:8020=. If the actual values are different, the job properties files in the examples directory must be edited to the correct values. The example applications are under the examples/app directory, one directory per example. The directory contains the @@ -108,7 +108,7 @@ import java.util.Properties; conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app"); . // setting workflow parameters - conf.setProperty("jobTracker", "foo:8021"); + conf.setProperty("resourceManager", "foo:8032"); conf.setProperty("inputDir", "/usr/tucu/inputdir"); conf.setProperty("outputDir", "/usr/tucu/outputdir"); ... @@ -159,7 +159,7 @@ import java.util.Properties; conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app"); . // setting workflow parameters - conf.setProperty("jobTracker", "foo:8021"); + conf.setProperty("resourceManager", "foo:8032"); conf.setProperty("inputDir", "/usr/tucu/inputdir"); conf.setProperty("outputDir", "/usr/tucu/outputdir"); ... @@ -183,6 +183,9 @@ import java.util.Properties; ... </verbatim> +Also asynchronous actions like FS action can be used / tested using =LocalOozie= / =OozieClient= API. Please see the module +=oozie-mini= for details like =fs-decision.xml= workflow example. + [[index][::Go back to Oozie Documentation Index::]] </noautolink> http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_Hive2ActionExtension.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_Hive2ActionExtension.twiki b/docs/src/site/twiki/DG_Hive2ActionExtension.twiki index 094307f..efbe56d 100644 --- a/docs/src/site/twiki/DG_Hive2ActionExtension.twiki +++ b/docs/src/site/twiki/DG_Hive2ActionExtension.twiki @@ -15,10 +15,8 @@ The =hive2= action runs Beeline to connect to Hive Server 2. The workflow job will wait until the Hive Server 2 job completes before continuing to the next action. -To run the Hive Server 2 job, you have to configure the =hive2= action with the -=job-tracker=, =name-node=, =jdbc-url=, =password= elements, and either -Hive's =script= or =query= element, as well as the necessary parameters -and configuration. +To run the Hive Server 2 job, you have to configure the =hive2= action with the =resource-manager=, =name-node=, =jdbc-url=, + =password= elements, and either Hive's =script= or =query= element, as well as the necessary parameters and configuration. A =hive2= action can be configured to create or delete HDFS directories before starting the Hive Server 2 job. @@ -38,11 +36,11 @@ syntax is =${VARIABLES}=. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.4"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> - <hive2 xmlns="uri:oozie:hive2-action:0.1"> - <job-tracker>[JOB-TRACKER]</job-tracker> + <hive2 xmlns="uri:oozie:hive2-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> <name-node>[NAME-NODE]</name-node> <prepare> <delete path="[PATH]"/> @@ -115,11 +113,11 @@ expressions. *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.4"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="my-hive2-action"> - <hive2 xmlns="uri:oozie:hive2-action:0.1"> - <job-tracker>foo:8021</job-tracker> + <hive2 xmlns="uri:oozie:hive2-action:1.0"> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${jobOutput}"/> @@ -155,6 +153,43 @@ with a Kerberized Hive Server 2. ---+++ AE.A Appendix A, Hive 2 XML-Schema +---++++ Hive 2 Action Schema Version 1.0 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive2="uri:oozie:hive2-action:1.0" elementFormDefault="qualified" + targetNamespace="uri:oozie:hive2-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="hive2" type="hive2:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="hive2:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</verbatim> + ---++++ Hive 2 Action Schema Version 0.2 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_HiveActionExtension.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_HiveActionExtension.twiki b/docs/src/site/twiki/DG_HiveActionExtension.twiki index a0d7b57..aaa74fa 100644 --- a/docs/src/site/twiki/DG_HiveActionExtension.twiki +++ b/docs/src/site/twiki/DG_HiveActionExtension.twiki @@ -16,9 +16,8 @@ The =hive= action runs a Hive job. The workflow job will wait until the Hive job completes before continuing to the next action. -To run the Hive job, you have to configure the =hive= action with the -=job-tracker=, =name-node= and Hive =script= (or Hive =query=) elements as -well as the necessary parameters and configuration. +To run the Hive job, you have to configure the =hive= action with the =resource-manager=, =name-node= and Hive =script= +(or Hive =query=) elements as well as the necessary parameters and configuration. A =hive= action can be configured to create or delete HDFS directories before starting the Hive job. @@ -30,7 +29,7 @@ Oozie EL expressions can be used in the inline configuration. Property values specified in the =configuration= element override values specified in the =job-xml= file. -Note that Hadoop =mapred.job.tracker= and =fs.default.name= properties +Note that YARN =yarn.resourcemanager.address= (=resource-manager=) and HDFS =fs.default.name= (=name-node=) properties must not be present in the inline configuration. As with Hadoop =map-reduce= jobs, it is possible to add files and @@ -44,11 +43,11 @@ syntax is =${VARIABLES}=. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> - <hive xmlns="uri:oozie:hive-action:0.2"> - <job-tracker>[JOB-TRACKER]</job-tracker> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> <name-node>[NAME-NODE]</name-node> <prepare> <delete path="[PATH]"/> @@ -109,11 +108,11 @@ expressions. *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="myfirsthivejob"> - <hive xmlns="uri:oozie:hive-action:0.2"> - <job-tracker>foo:8021</job-tracker> + <hive xmlns="uri:oozie:hive-action:1.0"> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${jobOutput}"/> @@ -154,6 +153,42 @@ property =oozie.hive.log.level=. The default value is =INFO=. ---+++ AE.A Appendix A, Hive XML-Schema +---++++ Hive Action Schema Version 1.0 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:hive="uri:oozie:hive-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:hive-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="hive" type="hive:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</verbatim> + ---++++ Hive Action Schema Version 0.6 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_ShellActionExtension.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_ShellActionExtension.twiki b/docs/src/site/twiki/DG_ShellActionExtension.twiki index 493254a..5894c28 100644 --- a/docs/src/site/twiki/DG_ShellActionExtension.twiki +++ b/docs/src/site/twiki/DG_ShellActionExtension.twiki @@ -30,7 +30,7 @@ Oozie EL expressions can be used in the inline configuration. Property values specified in the =configuration= element override values specified in the =job-xml= file. -Note that Hadoop =mapred.job.tracker= and =fs.default.name= properties +Note that YARN =yarn.resourcemanager.address= (=resource-manager=) and HDFS =fs.default.name= (=name-node=) properties must not be present in the inline configuration. As with Hadoop =map-reduce= jobs, it is possible to add files and @@ -48,11 +48,11 @@ command must follow the following requirements: *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.3"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> - <shell xmlns="uri:oozie:shell-action:0.1"> - <job-tracker>[JOB-TRACKER]</job-tracker> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> <name-node>[NAME-NODE]</name-node> <prepare> <delete path="[PATH]"/> @@ -129,11 +129,11 @@ expressions. How to run any shell script or perl script or CPP executable <verbatim> -<workflow-app xmlns='uri:oozie:workflow:0.3' name='shell-wf'> +<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'> <start to='shell1' /> <action name='shell1'> - <shell xmlns="uri:oozie:shell-action:0.1"> - <job-tracker>${jobTracker}</job-tracker> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -169,7 +169,7 @@ EXEC=script.sh #Perl script #EXEC=script.pl -jobTracker=localhost:8021 +resourceManager=localhost:8032 nameNode=hdfs://localhost:8020 queueName=default @@ -178,11 +178,11 @@ queueName=default How to run any java program bundles in a jar. <verbatim> -<workflow-app xmlns='uri:oozie:workflow:0.3' name='shell-wf'> +<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'> <start to='shell1' /> <action name='shell1'> - <shell xmlns="uri:oozie:shell-action:0.1"> - <job-tracker>${jobTracker}</job-tracker> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -214,7 +214,7 @@ oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script #Hello.jar file is expected to be in the Workflow directory. EXEC=Hello.jar -jobTracker=localhost:8021 +resourceManager=localhost:8032 nameNode=hdfs://localhost:8020 queueName=default </verbatim> @@ -251,6 +251,40 @@ Although Shell action can execute any shell command, there are some limitations. ---+++ AE.A Appendix A, Shell XML-Schema +---++++ Shell Action Schema Version 1.0 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:shell="uri:oozie:shell-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="shell" type="shell:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="shell:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</verbatim> + ---++++ Shell Action Schema Version 0.3 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_SparkActionExtension.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_SparkActionExtension.twiki b/docs/src/site/twiki/DG_SparkActionExtension.twiki index 294e7dd..66076ff 100644 --- a/docs/src/site/twiki/DG_SparkActionExtension.twiki +++ b/docs/src/site/twiki/DG_SparkActionExtension.twiki @@ -16,7 +16,7 @@ The workflow job will wait until the Spark job completes before continuing to the next action. To run the Spark job, you have to configure the =spark= action with -the =job-tracker=, =name-node=, Spark =master= elements as +the =resource-manager=, =name-node=, Spark =master= elements as well as the necessary elements, arguments and configuration. Spark options can be specified in an element called =spark-opts=. @@ -31,11 +31,11 @@ in the =job-xml= file. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.3"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> - <spark xmlns="uri:oozie:spark-action:0.1"> - <job-tracker>[JOB-TRACKER]</job-tracker> + <spark xmlns="uri:oozie:spark-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> <name-node>[NAME-NODE]</name-node> <prepare> <delete path="[PATH]"/> @@ -126,11 +126,11 @@ expressions. *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="myfirstsparkjob"> - <spark xmlns="uri:oozie:spark-action:0.1"> - <job-tracker>foo:8021</job-tracker> + <spark xmlns="uri:oozie:spark-action:1.0"> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${jobOutput}"/> @@ -184,11 +184,11 @@ For more information, please refer to [[AG_Install#Oozie_Share_Lib][installation *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> .... <action name="myfirstpysparkjob"> - <spark xmlns="uri:oozie:spark-action:0.1"> - <job-tracker>foo:8021</job-tracker> + <spark xmlns="uri:oozie:spark-action:1.0"> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${jobOutput}"/> @@ -229,7 +229,7 @@ Specifying relative path for symlink: Make sure that the file is within the application directory i.e. =oozie.wf.application.path= . <verbatim> - <spark xmlns="uri:oozie:spark-action:0.2"> + <spark xmlns="uri:oozie:spark-action:1.0"> ... <jar>py-spark-example-symlink.py</jar> ... @@ -241,7 +241,7 @@ Make sure that the file is within the application directory i.e. =oozie.wf.appli Specifying full path for symlink: <verbatim> - <spark xmlns="uri:oozie:spark-action:0.2"> + <spark xmlns="uri:oozie:spark-action:1.0"> ... <jar>spark-example-symlink.jar</jar> ... @@ -257,18 +257,54 @@ Specifying full path for symlink: ---+++ AE.A Appendix A, Spark XML-Schema ----++++ Spark Action Schema Version 0.1 +---++++ Spark Action Schema Version 1.0 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; - xmlns:spark="uri:oozie:spark-action:0.1" elementFormDefault="qualified" - targetNamespace="uri:oozie:spark-action:0.1"> + xmlns:spark="uri:oozie:spark-action:1.0" elementFormDefault="qualified" + targetNamespace="uri:oozie:spark-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="spark" type="spark:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="spark:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</verbatim> + +---++++ Spark Action Schema Version 0.2 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:spark="uri:oozie:spark-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:spark-action:0.2"> <xs:element name="spark" type="spark:ACTION"/> <xs:complexType name="ACTION"> <xs:sequence> - <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> - <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/> <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/> @@ -279,6 +315,8 @@ Specifying full path for symlink: <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/> <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> </xs:complexType> @@ -314,18 +352,18 @@ Specifying full path for symlink: </xs:schema> </verbatim> ----++++ Spark Action Schema Version 0.2 +---++++ Spark Action Schema Version 0.1 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; - xmlns:spark="uri:oozie:spark-action:0.2" elementFormDefault="qualified" - targetNamespace="uri:oozie:spark-action:0.2"> + xmlns:spark="uri:oozie:spark-action:0.1" elementFormDefault="qualified" + targetNamespace="uri:oozie:spark-action:0.1"> <xs:element name="spark" type="spark:ACTION"/> <xs:complexType name="ACTION"> <xs:sequence> - <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> - <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/> <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/> @@ -336,8 +374,6 @@ Specifying full path for symlink: <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/> <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/> <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> - <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> - <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> </xs:complexType> http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_SqoopActionExtension.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_SqoopActionExtension.twiki b/docs/src/site/twiki/DG_SqoopActionExtension.twiki index 906ba54..0317d0c 100644 --- a/docs/src/site/twiki/DG_SqoopActionExtension.twiki +++ b/docs/src/site/twiki/DG_SqoopActionExtension.twiki @@ -17,9 +17,8 @@ The =sqoop= action runs a Sqoop job. The workflow job will wait until the Sqoop job completes before continuing to the next action. -To run the Sqoop job, you have to configure the =sqoop= action with the -=job-tracker=, =name-node= and Sqoop =command= or =arg= elements as -well as configuration. +To run the Sqoop job, you have to configure the =sqoop= action with the =resource-manager=, =name-node= and Sqoop =command= +or =arg= elements as well as configuration. A =sqoop= action can be configured to create or delete HDFS directories before starting the Sqoop job. @@ -31,8 +30,8 @@ Oozie EL expressions can be used in the inline configuration. Property values specified in the =configuration= element override values specified in the =job-xml= file. -Note that Hadoop =mapred.job.tracker= and =fs.default.name= properties -must not be present in the inline configuration. +Note that YARN =yarn.resourcemanager.address= / =resource-manager= and HDFS =fs.default.name= / =name-node= properties must not +be present in the inline configuration. As with Hadoop =map-reduce= jobs, it is possible to add files and archives in order to make them available to the Sqoop job. Refer to the @@ -42,11 +41,11 @@ section for more information about this feature. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> - <sqoop xmlns="uri:oozie:sqoop-action:0.2"> - <job-tracker>[JOB-TRACKER]</job-tracker> + <sqoop xmlns="uri:oozie:sqoop-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> <name-node>[NAME-NODE]</name-node> <prepare> <delete path="[PATH]"/> @@ -108,11 +107,11 @@ expressions. Using the =command= element: <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="myfirsthivejob"> - <sqoop xmlns="uri:oozie:sqoop-action:0.2"> - <job-tracker>foo:8021</job-tracker> + <sqoop xmlns="uri:oozie:sqoop-action:1.0"> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${jobOutput}"/> @@ -135,11 +134,11 @@ Using the =command= element: The same Sqoop action using =arg= elements: <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... - <action name="myfirsthivejob"> - <sqoop xmlns="uri:oozie:sqoop-action:0.2"> - <job-tracker>foo:8021</job-tracker> + <action name="myfirstsqoopjob"> + <sqoop xmlns="uri:oozie:sqoop-action:1.0"> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${jobOutput}"/> @@ -192,6 +191,40 @@ property =oozie.sqoop.log.level=. The default value is =INFO=. ---+++ AE.A Appendix A, Sqoop XML-Schema +---++++ Sqoop Action Schema Version 1.0 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; + xmlns:sqoop="uri:oozie:sqoop-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:sqoop-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="sqoop" type="sqoop:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="sqoop:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice> + <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/> + </xs:choice> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</verbatim> + ---++++ Sqoop Action Schema Version 0.3 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/DG_SshActionExtension.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/DG_SshActionExtension.twiki b/docs/src/site/twiki/DG_SshActionExtension.twiki index c2e33c9..99b96f7 100644 --- a/docs/src/site/twiki/DG_SshActionExtension.twiki +++ b/docs/src/site/twiki/DG_SshActionExtension.twiki @@ -30,7 +30,7 @@ user contains ls -a). *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> <ssh xmlns="uri:oozie:ssh-action:0.1"> @@ -71,7 +71,7 @@ The configuration of the =ssh= action can be parameterized (templatized) using E *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="myssjob"> <ssh xmlns="uri:oozie:ssh-action:0.1"> http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/ENG_MiniOozie.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/ENG_MiniOozie.twiki b/docs/src/site/twiki/ENG_MiniOozie.twiki index 2c39726..0b16289 100644 --- a/docs/src/site/twiki/ENG_MiniOozie.twiki +++ b/docs/src/site/twiki/ENG_MiniOozie.twiki @@ -54,7 +54,7 @@ $ mvn clean test ---++ Create Tests Using MiniOozie -MiniOozie is a junit test class to test Oozie applications such as workflow and coordinator. The test case +MiniOozie is a JUnit test class to test Oozie applications such as workflow and coordinator. The test case needs to extend from MiniOozieTestCase and does the same as the example class 'WorkflowTest.java' to create Oozie workflow application properties and workflow XML. The example file is under Oozie source tree: @@ -70,6 +70,10 @@ The test directories under MiniOozie are: * =minitest/src/test/java= : as test-source directory * =minitest/src/test/resources= : as test-resource directory + +Also asynchronous actions like FS action can be used / tested using =LocalOozie= / =OozieClient= API. +Please see =fs-decision.xml= workflow example. + [[index][::Go back to Oozie Documentation Index::]] </noautolink> http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki index 038c430..5df6024 100644 --- a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki +++ b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki @@ -5,7 +5,7 @@ ----- ---+!! Oozie Specification, a Hadoop Workflow System -*<center>(v3.1)</center>* +*<center>(v5.0)</center>* The goal of this document is to define a workflow engine system specialized in coordinating the execution of Hadoop Map/Reduce and Pig jobs. @@ -248,7 +248,7 @@ A workflow definition must have one =start= node. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <start to="[NODE-NAME]"/> ... @@ -260,7 +260,7 @@ The =to= attribute is the name of first workflow node to execute. *Example:* <verbatim> -<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> ... <start to="firstHadoopJob"/> ... @@ -282,7 +282,7 @@ A workflow definition must have one =end= node. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <end name="[NODE-NAME]"/> ... @@ -294,7 +294,7 @@ The =name= attribute is the name of the transition to do to end the workflow job *Example:* <verbatim> -<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> ... <end name="end"/> </workflow-app> @@ -315,7 +315,7 @@ A workflow definition may have zero or more =kill= nodes. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <kill name="[NODE-NAME]"> <message>[MESSAGE-TO-LOG]</message> @@ -333,7 +333,7 @@ A =kill= node does not have transition elements because it ends the workflow job *Example:* <verbatim> -<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> ... <kill name="killBecauseNoInput"> <message>Input unavailable</message> @@ -363,7 +363,7 @@ boolean value, =true= or =false=. For example: *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <decision name="[NODE-NAME]"> <switch> @@ -391,7 +391,7 @@ state if none of the predicates evaluates to true. *Example:* <verbatim> -<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> ... <decision name="mydecision"> <switch> @@ -423,7 +423,7 @@ the same =fork= node. *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <fork name="[FORK-NODE-NAME]"> <path start="[NODE-NAME]" /> @@ -446,7 +446,7 @@ fork arrive to the join node. *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <fork name="forking"> <path start="firstparalleljob"/> @@ -454,7 +454,7 @@ fork arrive to the join node. </fork> <action name="firstparallejob"> <map-reduce> - <job-tracker>foo:8021</job-tracker> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <job-xml>job1.xml</job-xml> </map-reduce> @@ -463,7 +463,7 @@ fork arrive to the join node. </action> <action name="secondparalleljob"> <map-reduce> - <job-tracker>foo:8021</job-tracker> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <job-xml>job2.xml</job-xml> </map-reduce> @@ -658,7 +658,7 @@ public class MyConfigClass implements OozieActionConfigurator { To use your config class in your MapReduce action, simply compile it into a jar, make the jar available to your action, and specify the class name in the =config-class= element (this requires at least schema 0.5): <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.5"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> <map-reduce> @@ -724,11 +724,11 @@ Pipe properties can be overridden by specifying them in the =job-xml= file or =c ---+++++ 3.2.2.5 Syntax <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.5"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> <map-reduce> - <job-tracker>[JOB-TRACKER]</job-tracker> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> <name-node>[NAME-NODE]</name-node> <prepare> <delete path="[PATH]"/> @@ -808,11 +808,11 @@ The =mapper= and =reducer= process for streaming jobs, should specify the execut *Example:* <verbatim> -<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="myfirstHadoopJob"> <map-reduce> - <job-tracker>foo:8021</job-tracker> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="hdfs://foo:8020/usr/tucu/output-data"/> @@ -850,11 +850,11 @@ the workflow job configuration when creating the workflow job. *Streaming Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="firstjob"> <map-reduce> - <job-tracker>foo:8021</job-tracker> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${output}"/> @@ -891,11 +891,11 @@ the workflow job configuration when creating the workflow job. *Pipes Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="firstjob"> <map-reduce> - <job-tracker>foo:8021</job-tracker> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${output}"/> @@ -930,7 +930,7 @@ The =pig= action starts a Pig job. The workflow job will wait until the pig job completes before continuing to the next action. -The =pig= action has to be configured with the job-tracker, name-node, pig script and the necessary parameters and +The =pig= action has to be configured with the resource-manager, name-node, pig script and the necessary parameters and configuration to run the Pig job. A =pig= action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before @@ -948,12 +948,54 @@ the precedence order is later values override earlier values. Inline property values can be parameterized (templatized) using EL expressions. -The Hadoop =mapred.job.tracker= and =fs.default.name= properties must not be present in the job-xml and inline +The YARN =yarn.resourcemanager.address= and HDFS =fs.default.name= properties must not be present in the job-xml and inline configuration. As with Hadoop map-reduce jobs, it is possible to add files and archives to be available to the Pig job, refer to section [#FilesArchives][Adding Files and Archives for the Job]. + +*Syntax for Pig actions in Oozie schema 1.0:* +<verbatim> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <pig> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[JOB-XML-FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <script>[PIG-SCRIPT]</script> + <param>[PARAM-VALUE]</param> + ... + <param>[PARAM-VALUE]</param> + <argument>[ARGUMENT-VALUE]</argument> + ... + <argument>[ARGUMENT-VALUE]</argument> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + </pig> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</verbatim> + *Syntax for Pig actions in Oozie schema 0.2:* <verbatim> <workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.2"> @@ -995,7 +1037,6 @@ section [#FilesArchives][Adding Files and Archives for the Job]. </workflow-app> </verbatim> - *Syntax for Pig actions in Oozie schema 0.1:* <verbatim> @@ -1162,7 +1203,7 @@ executed. Thus there is less chance of an error occurring while the =fs= action *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.5"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> <fs> @@ -1229,7 +1270,7 @@ supported; this action requires an argument for RF. *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.5"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="hdfscommands"> <fs> @@ -1298,7 +1339,7 @@ There can be several sub-workflows defined within a single workflow, each under *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> <sub-workflow> @@ -1333,7 +1374,7 @@ The configuration of the =sub-workflow= action can be parameterized (templatized *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="a"> <sub-workflow> @@ -1375,7 +1416,7 @@ Java applications are executed in the Hadoop cluster as map-reduce job with a si The workflow job will wait until the java application completes its execution before continuing to the next action. -The =java= action has to be configured with the job-tracker, name-node, main Java class, JVM options and arguments. +The =java= action has to be configured with the resource-manager, name-node, main Java class, JVM options and arguments. To indicate an =ok= action transition, the main Java class must complete gracefully the =main= method invocation. @@ -1406,8 +1447,8 @@ in unexpected behaviour. For repeated configuration properties later values over Inline property values can be parameterized (templatized) using EL expressions. -The Hadoop =mapred.job.tracker= (=job-tracker=) and =fs.default.name= (=name-node=) properties must not be present in -the =job-xml= and in the inline configuration. +The YARN =yarn.resourcemanager.address= (=resource-manager=) and HDFS =fs.default.name= (=name-node=) properties must not be present +in the =job-xml= and in the inline configuration. As with =map-reduce= and =pig= actions, it is possible to add files and archives to be available to the Java application. Refer to section [#FilesArchives][Adding Files and Archives for the Job]. @@ -1438,11 +1479,11 @@ application must consider the possibility of collisions with activity spawned by *Syntax:* <verbatim> -<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> ... <action name="[NODE-NAME]"> <java> - <job-tracker>[JOB-TRACKER]</job-tracker> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> <name-node>[NAME-NODE]</name-node> <prepare> <delete path="[PATH]"/> @@ -1508,11 +1549,11 @@ All the above elements can be parameterized (templatized) using EL expressions. *Example:* <verbatim> -<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"> +<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0"> ... <action name="myfirstjavajob"> <java> - <job-tracker>foo:8021</job-tracker> + <resource-manager>foo:8032</resource-manager> <name-node>bar:8020</name-node> <prepare> <delete path="${jobOutput}"/> @@ -1595,11 +1636,11 @@ By using properties Parameterized Workflow definition: <verbatim> -<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:0.1"> +<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:1.0"> ... <action name='firstjob'> <map-reduce> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -1629,7 +1670,7 @@ Parameterized Workflow definition: When submitting a workflow job for the workflow definition above, 3 workflow job properties must be specified: - * =jobTracker:= + * =resourceManager:= * =inputDir:= * =outputDir:= @@ -1643,7 +1684,7 @@ properties are actually specified (i.e. before the job is executed and fails). D The previous parameterized workflow definition with formal parameters: <verbatim> -<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:0.4"> +<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:1.0"> <parameters> <property> <name>inputDir</name> @@ -1656,7 +1697,7 @@ The previous parameterized workflow definition with formal parameters: ... <action name='firstjob'> <map-reduce> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -1895,11 +1936,11 @@ This function can also be used to access specific action statistics information. Below is the workflow that describes how to access specific information using hadoop:counters() EL function from the MR stats. *Workflow xml:* <verbatim> -<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf"> +<workflow-app xmlns="uri:oozie:workflow:1.0" name="map-reduce-wf"> <start to="mr-node"/> <action name="mr-node"> <map-reduce> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <prepare> <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/> @@ -1940,7 +1981,7 @@ Below is the workflow that describes how to access specific information using ha </action> <action name="java1"> <java> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -2014,11 +2055,11 @@ Below is the workflow that describes how to access specific information using ha Below is the workflow that describes how to access specific information using hadoop:counters() EL function from the Pig stats. *Workflow xml:* <verbatim> -<workflow-app xmlns="uri:oozie:workflow:0.2" name="pig-wf"> +<workflow-app xmlns="uri:oozie:workflow:1.0" name="pig-wf"> <start to="pig-node"/> <action name="pig-node"> <pig> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <prepare> <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/pig"/> @@ -2046,7 +2087,7 @@ Below is the workflow that describes how to access specific information using ha </action> <action name="java1"> <java> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -2084,7 +2125,7 @@ hadoop configuration object directly, and then return the value of target proper <verbatim> <action name="mr-node"> <map-reduce> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -2104,11 +2145,11 @@ The function _wf:actionData()_ can be used to access Hadoop ID's for actions suc An example is shown below. <verbatim> -<workflow-app xmlns="uri:oozie:workflow:0.2" name="pig-wf"> +<workflow-app xmlns="uri:oozie:workflow:1.0" name="pig-wf"> <start to="pig-node"/> <action name="pig-node"> <pig> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <prepare> <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/pig"/> @@ -2132,7 +2173,7 @@ An example is shown below. </action> <action name="java1"> <java> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -2291,7 +2332,7 @@ specify an uber jar is governed by the =oozie.action.mapreduce.uber.jar.enable= <verbatim> <action name="mr-node"> <map-reduce> - <job-tracker>${jobTracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${nameNode}</name-node> <configuration> <property> @@ -2549,7 +2590,7 @@ actions in workflow xml if needed. The =retry-interval= should be specified in m Examples of User-Retry in a workflow action is : <verbatim> -<workflow-app xmlns="uri:oozie:workflow:0.5" name="wf-name"> +<workflow-app xmlns="uri:oozie:workflow:1.0" name="wf-name"> <action name="a" retry-max="2" retry-interval="1" retry-policy="exponential"> </action> </verbatim> @@ -2557,17 +2598,17 @@ Examples of User-Retry in a workflow action is : #GlobalConfigurations ---++ 19 Global Configurations -Oozie allows a global section to reduce the redundant job-tracker and name-node declarations for each action. The user can define -a =global= section in the beginning of the workflow.xml . The global section may contain the job-xml, configuration, job-tracker, or -name-node that the user would like to set for every action. If a user then redefines one of these in a specific action node, -Oozie will update use the specific declaration instead of the global one for that action. +Oozie allows a global section to reduce the redundant resource-manager and name-node declarations for each action. The user can +define a =global= section in the beginning of the =workflow.xml=. The global section may contain the =job-xml=, =configuration=, +=resource-manager=, or =name-node= that the user would like to set for every action. If a user then redefines one of these in a +specific action node, Oozie will update use the specific declaration instead of the global one for that action. Example of a global element: <verbatim> -<workflow-app xmlns="uri:oozie:workflow:0.4" name="wf-name"> +<workflow-app xmlns="uri:oozie:workflow:1.0" name="wf-name"> <global> - <job-tracker>${job-tracker}</job-tracker> + <resource-manager>${resourceManager}</resource-manager> <name-node>${namd-node}</name-node> <job-xml>job1.xml</job-xml> <configuration> @@ -2599,8 +2640,376 @@ to be executed. ---++ Appendixes #OozieWFSchema ----+++ Appendix A, Oozie XML-Schema ----++++ Oozie Schema Version 0.5 +---+++ Appendix A, Oozie Workflow and Common XML Schemas + +---++++ Oozie Workflow Schema Version 1.0 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:1.0" + elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="workflow-app" type="workflow:WORKFLOW-APP"/> +. + <xs:simpleType name="IDENTIFIER"> + <xs:restriction base="xs:string"> + <xs:pattern value="([a-zA-Z_]([\-_a-zA-Z0-9])*){1,39}"/> + </xs:restriction> + </xs:simpleType> +. + <xs:complexType name="WORKFLOW-APP"> + <xs:sequence> + <xs:element name="parameters" type="workflow:PARAMETERS" minOccurs="0" maxOccurs="1"/> + <xs:element name="global" type="workflow:GLOBAL" minOccurs="0" maxOccurs="1"/> + <xs:element name="credentials" type="workflow:CREDENTIALS" minOccurs="0" maxOccurs="1"/> + <xs:element name="start" type="workflow:START" minOccurs="1" maxOccurs="1"/> + <xs:choice minOccurs="0" maxOccurs="unbounded"> + <xs:element name="decision" type="workflow:DECISION" minOccurs="1" maxOccurs="1"/> + <xs:element name="fork" type="workflow:FORK" minOccurs="1" maxOccurs="1"/> + <xs:element name="join" type="workflow:JOIN" minOccurs="1" maxOccurs="1"/> + <xs:element name="kill" type="workflow:KILL" minOccurs="1" maxOccurs="1"/> + <xs:element name="action" type="workflow:ACTION" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="end" type="workflow:END" minOccurs="1" maxOccurs="1"/> + <xs:any namespace="uri:oozie:sla:0.1 uri:oozie:sla:0.2" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="PARAMETERS"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="0" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="GLOBAL"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="workflow:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="workflow:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="START"> + <xs:attribute name="to" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> + + <xs:complexType name="END"> + <xs:attribute name="name" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:complexType name="DECISION"> + <xs:sequence> + <xs:element name="switch" type="workflow:SWITCH" minOccurs="1" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:element name="switch" type="workflow:SWITCH"/> + + <xs:complexType name="SWITCH"> + <xs:sequence> + <xs:sequence> + <xs:element name="case" type="workflow:CASE" minOccurs="1" maxOccurs="unbounded"/> + <xs:element name="default" type="workflow:DEFAULT" minOccurs="1" maxOccurs="1"/> + </xs:sequence> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CASE"> + <xs:simpleContent> + <xs:extension base="xs:string"> + <xs:attribute name="to" type="workflow:IDENTIFIER" use="required"/> + </xs:extension> + </xs:simpleContent> + </xs:complexType> +. + <xs:complexType name="DEFAULT"> + <xs:attribute name="to" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:complexType name="FORK_TRANSITION"> + <xs:attribute name="start" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:complexType name="FORK"> + <xs:sequence> + <xs:element name="path" type="workflow:FORK_TRANSITION" minOccurs="2" maxOccurs="unbounded"/> + </xs:sequence> + <xs:attribute name="name" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:complexType name="JOIN"> + <xs:attribute name="name" type="workflow:IDENTIFIER" use="required"/> + <xs:attribute name="to" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:element name="kill" type="workflow:KILL"/> +. + <xs:complexType name="KILL"> + <xs:sequence> + <xs:element name="message" type="xs:string" minOccurs="1" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:complexType name="ACTION_TRANSITION"> + <xs:attribute name="to" type="workflow:IDENTIFIER" use="required"/> + </xs:complexType> +. + <xs:element name="map-reduce" type="workflow:MAP-REDUCE"/> + <xs:element name="pig" type="workflow:PIG"/> + <xs:element name="sub-workflow" type="workflow:SUB-WORKFLOW"/> + <xs:element name="fs" type="workflow:FS"/> + <xs:element name="java" type="workflow:JAVA"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice minOccurs="1" maxOccurs="1"> + <xs:element name="map-reduce" type="workflow:MAP-REDUCE" minOccurs="1" maxOccurs="1"/> + <xs:element name="pig" type="workflow:PIG" minOccurs="1" maxOccurs="1"/> + <xs:element name="sub-workflow" type="workflow:SUB-WORKFLOW" minOccurs="1" maxOccurs="1"/> + <xs:element name="fs" type="workflow:FS" minOccurs="1" maxOccurs="1"/> + <xs:element name="java" type="workflow:JAVA" minOccurs="1" maxOccurs="1"/> + <xs:any namespace="##other" minOccurs="1" maxOccurs="1"/> + </xs:choice> + <xs:element name="ok" type="workflow:ACTION_TRANSITION" minOccurs="1" maxOccurs="1"/> + <xs:element name="error" type="workflow:ACTION_TRANSITION" minOccurs="1" maxOccurs="1"/> + <xs:any namespace="uri:oozie:sla:0.1 uri:oozie:sla:0.2" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="workflow:IDENTIFIER" use="required"/> + <xs:attribute name="cred" type="xs:string"/> + <xs:attribute name="retry-max" type="xs:string"/> + <xs:attribute name="retry-interval" type="xs:string"/> + <xs:attribute name="retry-policy" type="xs:string"/> + </xs:complexType> +. + <xs:complexType name="MAP-REDUCE"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="workflow:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="0" maxOccurs="1"> + <xs:element name="streaming" type="workflow:STREAMING" minOccurs="0" maxOccurs="1"/> + <xs:element name="pipes" type="workflow:PIPES" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="launcher" type="workflow:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="workflow:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="config-class" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PIG"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="workflow:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="workflow:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="workflow:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="SUB-WORKFLOW"> + <xs:sequence> + <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="propagate-configuration" type="workflow:FLAG" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="workflow:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="FS"> + <xs:sequence> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="workflow:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:choice minOccurs="0" maxOccurs="unbounded"> + <xs:element name="delete" type="workflow:DELETE"/> + <xs:element name="mkdir" type="workflow:MKDIR"/> + <xs:element name="move" type="workflow:MOVE"/> + <xs:element name="chmod" type="workflow:CHMOD"/> + <xs:element name="touchz" type="workflow:TOUCHZ"/> + <xs:element name="chgrp" type="workflow:CHGRP"/> + </xs:choice> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="JAVA"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="workflow:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="workflow:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="workflow:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="main-class" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:choice minOccurs="0" maxOccurs="1"> + <xs:element name="java-opts" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="java-opt" type="xs:string" minOccurs="1" maxOccurs="unbounded"/> + </xs:choice> + <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="workflow:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="STREAMING"> + <xs:sequence> + <xs:element name="mapper" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="reducer" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="record-reader" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="record-reader-mapping" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PIPES"> + <xs:sequence> + <xs:element name="map" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="reduce" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="inputformat" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="partitioner" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="writer" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="program" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CREDENTIALS"> + <xs:sequence minOccurs="0" maxOccurs="unbounded"> + <xs:element name="credential" type="workflow:CREDENTIAL"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="CREDENTIAL"> + <xs:sequence minOccurs="0" maxOccurs="unbounded" > + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + <xs:attribute name="name" type="xs:string" use="required"/> + <xs:attribute name="type" type="xs:string" use="required"/> + </xs:complexType> +. +</xs:schema> +</verbatim> + +---++++ Oozie Common Schema Version 1.0 +<verbatim> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; elementFormDefault="qualified"> +. + <xs:complexType name="LAUNCHER"> + <xs:choice maxOccurs="unbounded"> + <!-- Oozie Launcher job memory in MB --> + <xs:element name="memory.mb" minOccurs="0" type="xs:unsignedInt"/> + <xs:element name="vcores" minOccurs="0" type="xs:unsignedInt"/> + <xs:element name="java-opts" minOccurs="0" type="xs:string"/> + <xs:element name="env" minOccurs="0" type="xs:string"/> + <xs:element name="queue" minOccurs="0" type="xs:string"/> + <xs:element name="sharelib" minOccurs="0" type="xs:string"/> + <xs:element name="view-acl" minOccurs="0" type="xs:string"/> + <xs:element name="modify-acl" minOccurs="0" type="xs:string"/> + </xs:choice> + </xs:complexType> +. + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> +. + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="FLAG"/> +. + <xs:complexType name="MOVE"> + <xs:attribute name="source" type="xs:string" use="required"/> + <xs:attribute name="target" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="CHMOD"> + <xs:sequence> + <xs:element name="recursive" type="FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="path" type="xs:string" use="required"/> + <xs:attribute name="permissions" type="xs:string" use="required"/> + <xs:attribute name="dir-files" type="xs:string"/> + </xs:complexType> +. + <xs:complexType name="TOUCHZ"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> +. + <xs:complexType name="CHGRP"> + <xs:sequence> + <xs:element name="recursive" type="FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="path" type="xs:string" use="required"/> + <xs:attribute name="group" type="xs:string" use="required"/> + <xs:attribute name="dir-files" type="xs:string"/> + </xs:complexType> +. +</xs:schema> +</verbatim> + +---++++ Oozie Workflow Schema Version 0.5 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:0.5" elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:0.5"> @@ -2928,7 +3337,7 @@ to be executed. </xs:schema> </verbatim> ----++++ Oozie Schema Version 0.4.5 +---++++ Oozie Workflow Schema Version 0.4.5 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:0.4.5" elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:0.4.5"> @@ -3253,7 +3662,7 @@ to be executed. </xs:schema> </verbatim> ----++++ Oozie Schema Version 0.4 +---++++ Oozie Workflow Schema Version 0.4 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:0.4" elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:0.4"> @@ -3561,7 +3970,7 @@ to be executed. </xs:schema> </verbatim> ----++++ Oozie Schema Version 0.3 +---++++ Oozie Workflow Schema Version 0.3 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:0.3" elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:0.3"> @@ -3837,7 +4246,7 @@ to be executed. </xs:schema> </verbatim> ----++++ Oozie Schema Version 0.2.5 +---++++ Oozie Workflow Schema Version 0.2.5 <verbatim> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:0.2.5" elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:0.2.5"> @@ -4109,7 +4518,7 @@ to be executed. </xs:schema> </verbatim> ----++++ Oozie Schema Version 0.2 +---++++ Oozie Workflow Schema Version 0.2 <verbatim> <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:0.2" @@ -4463,7 +4872,7 @@ to be executed. </xs:schema> </verbatim> ----++++ Oozie Schema Version 0.1 +---++++ Oozie Workflow Schema Version 0.1 <verbatim> <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"; xmlns:workflow="uri:oozie:workflow:0.1" @@ -4697,17 +5106,17 @@ The output of the jobs in the previous step are use as input for the next jobs. *Required workflow job parameters:* - * =jobtracker= : JobTracker HOST:PORT + * =resourcemanager= : ResourceManager HOST:PORT * =namenode= : NameNode HOST:PORT * =input= : input directory * =output= : output directory <verbatim> -<workflow-app name='example-forkjoinwf' xmlns="uri:oozie:workflow:0.1"> +<workflow-app name='example-forkjoinwf' xmlns="uri:oozie:workflow:1.0"> <start to='firstjob' /> <action name="firstjob"> <map-reduce> - <job-tracker>${jobtracker}</job-tracker> + <resource-manager>${resourcemanager}</resource-manager> <name-node>${namenode}</name-node> <configuration> <property> @@ -4741,7 +5150,7 @@ The output of the jobs in the previous step are use as input for the next jobs. </fork> <action name="secondjob"> <map-reduce> - <job-tracker>${jobtracker}</job-tracker> + <resource-manager>${resourcemanager}</resource-manager> <name-node>${namenode}</name-node> <configuration> <property> @@ -4771,7 +5180,7 @@ The output of the jobs in the previous step are use as input for the next jobs. </action> <action name="thirdjob"> <map-reduce> - <job-tracker>${jobtracker}</job-tracker> + <resource-manager>${resourcemanager}</resource-manager> <name-node>${namenode}</name-node> <configuration> <property> @@ -4802,7 +5211,7 @@ The output of the jobs in the previous step are use as input for the next jobs. <join name='join' to='finalejob'/> <action name="finaljob"> <map-reduce> - <job-tracker>${jobtracker}</job-tracker> + <resource-manager>${resourcemanager}</resource-manager> <name-node>${namenode}</name-node> <configuration> <property> http://git-wip-us.apache.org/repos/asf/oozie/blob/50e6c35b/release-log.txt ---------------------------------------------------------------------- diff --git a/release-log.txt b/release-log.txt index 6be3de2..76401f5 100644 --- a/release-log.txt +++ b/release-log.txt @@ -1,5 +1,6 @@ -- Oozie 5.0.0 release (trunk - unreleased) +OOZIE-2600 OYA: Update Documentation (andras.piros via gezapeti) OOZIE-3189 Update the release script and wiki page to use sha512 instead of md5 (rkanter via gezapeti) OOZIE-3195 Typo in WebServicesAPI.twiki: Proxy Hive Job Submission (kmarton via andras.piros) OOZIE-2814 OYA: Update example workflows to newest schemas (asasvari via gezapeti)
