Added: oozie/site/trunk/content/resources/docs/5.2.0/DG_SLAMonitoring.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.0/DG_SLAMonitoring.html?rev=1870914&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.0/DG_SLAMonitoring.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.0/DG_SLAMonitoring.html Fri Dec 6 08:59:57 2019 @@ -0,0 +1,971 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2019-12-05 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20191205" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/"; class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.0"> + 5.2.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-12-05</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Oozie SLA Monitoring</h1> +<ul> +<li><a href="#Overview">Overview</a></li> +<li><a href="#Oozie_Server_Configuration">Oozie Server Configuration</a></li> +<li><a href="#SLA_Tracking">SLA Tracking</a> +<ul> +<li><a href="#Event_Status">Event Status</a></li> +<li><a href="#SLA_Status">SLA Status</a></li></ul></li> +<li><a href="#Configuring_SLA_in_Applications">Configuring SLA in Applications</a> +<ul> +<li><a href="#SLA_Definition_in_Workflow">SLA Definition in Workflow</a></li> +<li><a href="#SLA_Definition_in_Workflow_Action">SLA Definition in Workflow Action</a></li> +<li><a href="#SLA_Definition_in_Coordinator_Action">SLA Definition in Coordinator Action</a></li></ul></li> +<li><a href="#Accessing_SLA_Information">Accessing SLA Information</a> +<ul> +<li><a href="#Scenario_1:_Workflow_Job_Start_Miss">Scenario 1: Workflow Job Start_Miss</a></li> +<li><a href="#Scenario_2:_Workflow_Action_End_Miss">Scenario 2: Workflow Action End_Miss</a></li> +<li><a href="#Scenario_3:_Coordinator_Action_Duration_Miss">Scenario 3: Coordinator Action Duration_Miss</a></li> +<li><a href="#Scenario_4:_All_Coordinator_actions_in_a_Bundle">Scenario 4: All Coordinator actions in a Bundle</a></li> +<li><a href="#Scenario_5:_Workflow_jobs_actually_started_in_a_24_hour_period">Scenario 5: Workflow jobs actually started in a 24 hour period</a></li> +<li><a href="#Scenario_6:_Workflow_jobs_executed_much_faster_than_required">Scenario 6: Workflow jobs executed much faster than required</a></li> +<li><a href="#Scenario_7:_Coordinator_actions_with_START_MET_or_END_MET_event_status">Scenario 7: Coordinator actions with START_MET or END_MET event status</a></li> +<li><a href="#Scenario_8:_Not_yet_started_workflow_jobs_expected_to_start_before_a_specified_date.">Scenario 8: Not yet started workflow jobs expected to start before a specified date.</a></li> +<li><a href="#Scenario_9:_Filtering_for_app_name_using__wildchar">Scenario 9: Filtering for app_name using % wildchar</a></li> +<li><a href="#Scenario_9:_Filtering_for_app_name_using__wildchar_and_sorting_the_order_by_the_application_name_in_descending_order">Scenario 9: Filtering for app_name using % wildchar and sorting the order by the application name in descending order</a></li> +<li><a href="#Sample_Email_Alert">Sample Email Alert</a></li> +<li><a href="#Changing_job_SLA_definition_and_alerting">Changing job SLA definition and alerting</a> +<ul> +<li><a href="#a1._Specify_in_Bundle_XML_during_submission.">1. Specify in Bundle XML during submission.</a></li> +<li><a href="#a2._Specify_during_Coordinator_job_submission_or_update">2. Specify during Coordinator job submission or update</a></li> +<li><a href="#a3._Change_using_command_line">3. Change using command line</a></li> +<li><a href="#a4._Change_using_REST_API">4. Change using REST API</a></li></ul></li></ul></li> +<li><a href="#In-memory_SLA_entries_and_database_content">In-memory SLA entries and database content</a></li> +<li><a href="#Known_issues">Known issues</a></li></ul> + +<div class="section"> +<h2><a name="Overview"></a>Overview</h2> +<p>Critical jobs can have certain SLA requirements associated with them. This SLA can be in terms of time i.e. a maximum allowed time limit associated with when the job should start, by when should it end, and its duration of run. Oozie workflows and coordinators allow defining such SLA limits in the application definition xml.</p> +<p>With the addition of SLA Monitoring, Oozie can now actively monitor the state of these SLA-sensitive jobs and send out notifications for SLA mets and misses.</p> +<p>In versions earlier than 4.x, this was a passive feature where users needed to query the Oozie client SLA API to fetch the records regarding job status changes, and use their own custom calculation engine to compute whether SLA was met or missed, based on initial definition of time limits.</p> +<p>Oozie now also has a SLA tab in the Oozie UI, where users can query for SLA information and have a summarized view of how their jobs fared against their SLAs.</p></div> +<div class="section"> +<h2><a name="Oozie_Server_Configuration"></a>Oozie Server Configuration</h2> +<p>Refer to <a href="AG_Install.html#Notifications_Configuration">Notifications Configuration</a> for configuring Oozie server to track SLA for jobs and send notifications.</p></div> +<div class="section"> +<h2><a name="SLA_Tracking"></a>SLA Tracking</h2> +<p>Oozie allows tracking SLA for meeting the following criteria:</p> +<ul> + +<li>Start time</li> +<li>End time</li> +<li>Job Duration</li> +</ul> +<div class="section"> +<h3><a name="Event_Status"></a>Event Status</h3> +<p>Corresponding to each of these 3 criteria, your jobs are processed for whether Met or Miss i.e.</p> +<ul> + +<li>START_MET, START_MISS</li> +<li>END_MET, END_MISS</li> +<li>DURATION_MET, DURATION_MISS</li> +</ul></div> +<div class="section"> +<h3><a name="SLA_Status"></a>SLA Status</h3> +<p>Expected end-time is the most important criterion for majority of users while deciding overall SLA Met or Miss. Hence the <i>“SLA_Status”</i> for a job will transition through these four stages</p> +<ul> + +<li>Not_Started <– Job not yet begun</li> +<li>In_Process <– Job started and is running, and SLAs are being tracked</li> +<li>Met <– caused by an END_MET</li> +<li>Miss <– caused by an END_MISS</li> +</ul> +<p>In addition to overshooting expected end-time, and END_MISS (and so an eventual SLA MISS) also occurs when the job does not end successfully e.g. goes to error state - Failed/Killed/Error/Timedout.</p></div></div> +<div class="section"> +<h2><a name="Configuring_SLA_in_Applications"></a>Configuring SLA in Applications</h2> +<p>To make your jobs trackable for SLA, you simply need to add the <tt><sla:info></tt> tag to your workflow application definition. If you were already using the existing SLA schema in your workflows (Schema xmlns:sla=“uri:oozie:sla:0.1”), you don’t need to do anything extra to receive SLA notifications via JMS messages. This new SLA monitoring framework is backward-compatible - no need to change application XML for now and you can continue to fetch old records via the <a href="DG_CommandLineTool.html#SLAOperations">command line API</a>. However, usage of old schema and API is deprecated and we strongly recommend using new schema.</p> +<ul> + +<li>New SLA schema is ‘uri:oozie:sla:0.2’</li> +<li>In order to use new SLA schema, you will need to upgrade your workflow/coordinator schema to 0.5 i.e. ‘uri:oozie:workflow:0.5’</li> +</ul> +<div class="section"> +<h3><a name="SLA_Definition_in_Workflow"></a>SLA Definition in Workflow</h3> +<p>Example:</p> + +<div> +<div> +<pre class="source"><workflow-app name="test-wf-job-sla" + xmlns="uri:oozie:workflow:0.5" + xmlns:sla="uri:oozie:sla:0.2"> + <start to="grouper"/> + <action name="grouper"> + <map-reduce> + <job-tracker>jt</job-tracker> + <name-node>nn</name-node> + <configuration> + <property> + <name>mapred.input.dir</name> + <value>input</value> + </property> + <property> + <name>mapred.output.dir</name> + <value>output</value> + </property> + </configuration> + </map-reduce> + <ok to="end"/> + <error to="end"/> + </action> + <end name="end"/> + <sla:info> + <sla:nominal-time>${nominal_time}</sla:nominal-time> + <sla:should-start>${10 * MINUTES}</sla:should-start> + <sla:should-end>${30 * MINUTES}</sla:should-end> + <sla:max-duration>${30 * MINUTES}</sla:max-duration> + <sla:alert-events>start_miss,end_miss,duration_miss</sla:alert-events> + <sla:alert-contact>[email protected]</sla:alert-contact> + </sla:info> +</workflow-app> +</pre></div></div> + +<p>For the list of tags usable under <tt><sla:info></tt>, refer to <a href="WorkflowFunctionalSpec.html#SLASchema">Schemas Appendix</a>. This new schema is much more compact and meaningful, getting rid of redundant and unused tags.</p> +<ul> + +<li><tt>nominal-time</tt>: As the name suggests, this is the time relative to which your jobs’ SLAs will be calculated. Generally since Oozie workflows are aligned with synchronous data dependencies, this nominal time can be parameterized to be passed the value of your coordinator nominal time. Nominal time is also required in case of independent workflows and you can specify the time in which you expect the workflow to be run if you don’t have a synchronous dataset associated with it.</li> +<li><tt>should-start</tt>: Relative to <tt>nominal-time</tt> this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should <i>start running</i> to meet SLA. This is optional.</li> +<li><tt>should-end</tt>: Relative to <tt>nominal-time</tt> this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should <i>finish</i> to meet SLA.</li> +<li><tt>max-duration</tt>: This is the maximum amount of time (along with time-unit - MINUTES, HOURS, DAYS) your job is expected to run. This is optional.</li> +<li><tt>alert-events</tt>: Specify the types of events for which <b>Email</b> alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not necessary. However, note that this setting is only for alerts via <i>email</i> alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured.</li> +<li><tt>alert-contact</tt>: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts.</li> +</ul> +<p>NOTE: All tags can be parameterized as a EL function or a fixed value.</p> +<p>Same schema can be applied to and embedded under Workflow-Action as well as Coordinator-Action XML.</p></div> +<div class="section"> +<h3><a name="SLA_Definition_in_Workflow_Action"></a>SLA Definition in Workflow Action</h3> + +<div> +<div> +<pre class="source"><workflow-app name="test-wf-action-sla" xmlns="uri:oozie:workflow:0.5" xmlns:sla="uri:oozie:sla:0.2"> + <start to="grouper"/> + <action name="grouper"> + ... + <ok to="end"/> + <error to="end"/> + <sla:info> + <sla:nominal-time>${nominal_time}</sla:nominal-time> + <sla:should-start>${10 * MINUTES}</sla:should-start> + ... + </sla:info> + </action> + <end name="end"/> +</workflow-app> +</pre></div></div> +</div> +<div class="section"> +<h3><a name="SLA_Definition_in_Coordinator_Action"></a>SLA Definition in Coordinator Action</h3> + +<div> +<div> +<pre class="source"><coordinator-app name="test-coord-sla" frequency="${coord:days(1)}" freq_timeunit="DAY" + end_of_duration="NONE" start="2013-06-20T08:01Z" end="2013-12-01T08:01Z" + timezone="America/Los_Angeles" xmlns="uri:oozie:coordinator:0.4" xmlns:sla="uri:oozie:sla:0.2"> + <action> + <workflow> + <app-path>${wfAppPath}</app-path> + </workflow> + <sla:info> + <sla:nominal-time>${nominal_time}</sla:nominal-time> + ... + </sla:info> + </action> +</coordinator-app> +</pre></div></div> +</div></div> +<div class="section"> +<h2><a name="Accessing_SLA_Information"></a>Accessing SLA Information</h2> +<p>SLA information is accessible via the following ways:</p> +<ul> + +<li>Through the SLA tab of the Oozie Web UI.</li> +<li>JMS messages sent to a configured JMS provider for instantaneous tracking.</li> +<li>RESTful API to query for SLA summary.</li> +<li>As an <tt>Instrumentation.Counter</tt> entry that is accessible via RESTful API and reflects to the number of all SLA tracked external entities. Name of this counter is <tt>sla-calculator.sla-map</tt>.</li> +</ul> +<p>For JMS Notifications, you have to have a message broker in place, on which Oozie publishes messages and you can hook on a subscriber to receive those messages. For more info on setting up and consuming JMS messages, refer <a href="DG_JMSNotifications.html">JMS Notifications</a> documentation.</p> +<p>In the REST API, the following filters can be applied while fetching SLA information:</p> +<ul> + +<li><tt>app_name</tt> - Application name</li> +<li><tt>id</tt> - id of the workflow job, workflow action or coordinator action</li> +<li><tt>parent_id</tt> - Parent id of the workflow job, workflow action or coordinator action</li> +<li><tt>nominal_after</tt> and <tt>nominal_before</tt> - Start and End range for nominal time of the workflow or coordinator.</li> +<li><tt>bundle</tt> - Bundle Job ID or Bundle App Name. Fetches SLA information for actions of all coordinators in that bundle.</li> +<li><tt>event_status</tt> - event status such as START_MET/START_MISS/DURATION_MET/DURATION_MISS/END_MET/END_MISS</li> +<li><tt>sla_status</tt> - sla status such as NOT_STARTED/IN_PROCESS/MET/MISS</li> +<li><tt>job_status</tt> - job status such as CREATED/STARTED/SUCCEEDED/KILLED/FAILED</li> +<li><tt>app_type</tt> - application type such as COORDINATOR_ACTION/COORDINATOR_JOB/WORKFLOW_JOB/WORKFLOW_ACTION</li> +<li><tt>user_name</tt> - the username of the user who submitted the job</li> +<li><tt>created_after</tt> and <tt>created_before</tt> - Start and End range for created time of the workflow or coordinator.</li> +<li><tt>expectedstart_after</tt> and <tt>expectedstart_before</tt> - Start and End range for expected start time of the workflow or coordinator.</li> +<li><tt>expectedend_after</tt> and <tt>expectedend_before</tt> - Start and End range for expected end time of the workflow or coordinator.</li> +<li><tt>actualstart_after</tt> and <tt>actualstart_before</tt> - Start and End range for actual start time of the workflow or coordinator.</li> +<li><tt>actualend_after</tt> and <tt>actualend_before</tt> - Start and End range for actual end time of the workflow or coordinator.</li> +<li><tt>actual_duration_min</tt> and <tt>actual_duration_max</tt> - Min and Max range for actual duration (in milliseconds)</li> +<li><tt>expected_duration_min</tt> and <tt>expected_duration_max</tt> - Min and Max range for expected duration (in milliseconds)</li> +</ul> +<p>It is possible to specify multiple filter conditions with semicolon separation, only information meeting all the conditions will be fetched.</p> +<p>Multiple <tt>event_status</tt> and <tt>sla_status</tt> can be specified with comma separation. When multiple statuses are specified, they are considered as OR. For example, <tt>event_status=START_MET,END_MISS</tt> list the coordinator actions where event status is either <tt>START_MET</tt> OR <tt>END_MISS</tt>.</p> +<p>For the <tt>app_name</tt>, <tt>app_type</tt>, <tt>user_name</tt>, and <tt>job_status</tt> filter fields two wildchars can also be used: the percent sign ( <tt>%</tt> ) represents zero, one, or multiple characters, the underscore ( <tt>_</tt> ) character represents a single character.</p> +<p>For compatibility reasons <tt>nominal_start</tt> and <tt>nominal_end</tt> filter names can also be used instead of <tt>nominal_after</tt> and <tt>nominal_before</tt>.</p> +<p>When <tt>timezone</tt> query parameter is specified, the expected and actual start/end time returned is formatted. If not specified, the number of milliseconds that have elapsed since January 1, 1970 00:00:00.000 GMT is returned.</p> +<p>It is possible to specify the ordering of the list by using the <tt>sortby</tt> and the <tt>order</tt> parameters. The possible values for the <tt>sortby</tt> parameter are: actualDuration, actualEndTS, actualStartTS, appName, appType, createdTimeTS, eventProcessed, eventStatus, expectedDuration, expectedEndTS, expectedStartTS, jobId, jobStatus, lastModifiedTS, nominalTimeTS, parentId, slaStatus, and user. The possible value for the <tt>order</tt> parameter are: desc and asc. The default value for the <tt>sortby</tt> parameter is nominalTimeTS, the default value for the <tt>order</tt> parameter is asc. If several items has the same sortby column values the these items will be sorted by the ascending order of the nominalTimeTS field.</p> +<p>The examples below demonstrate the use of REST API and explains the JSON response.</p> +<div class="section"> +<h3><a name="Scenario_1:_Workflow_Job_Start_Miss"></a>Scenario 1: Workflow Job Start_Miss</h3> +<p><b>Request:</b></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=nominal_after=2013-06-18T00:01Z;nominal_before=2013-06-23T00:01Z;app_name=my-sla-app +</pre></div></div> + +<p><b>JSON Response</b></p> + +<div> +<div> +<pre class="source">{ + + id : "000056-1238791320234-oozie-joe-W" + parentId : "000001-1238791320234-oozie-joe-C@8" + appType : "WORKFLOW_JOB" + msgType : "SLA" + appName : "my-sla-app" + slaStatus : "IN_PROCESS" + jobStatus : "RUNNING" + user: "joe" + nominalTime: "2013-16-22T05:00Z" + expectedStartTime: "2013-16-22T05:10Z" <-- (should start by this time) + actualStartTime: "2013-16-22T05:30Z" <-- (20 min late relative to expected start) + expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time) + actualEndTime: null + expectedDuration: 900000 <-- (expected duration in milliseconds) + actualDuration: 120000 <-- (actual duration in milliseconds) + notificationMessage: "My Job has encountered an SLA event!" + upstreamApps: "dependent-app-1, dependent-app-2" + +} +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Scenario_2:_Workflow_Action_End_Miss"></a>Scenario 2: Workflow Action End_Miss</h3> +<p><b>Request:</b></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=parent_id=000056-1238791320234-oozie-joe-W +</pre></div></div> + +<p><b>JSON Response</b></p> + +<div> +<div> +<pre class="source">{ + + id : "000056-1238791320234-oozie-joe-W@map-reduce-action" + parentId : "000056-1238791320234-oozie-joe-W" + appType : "WORKFLOW_ACTION" + msgType : "SLA" + appName : "map-reduce-action" + slaStatus : "MISS" + jobStatus : "SUCCEEDED" + user: "joe" + nominalTime: "2013-16-22T05:00Z" + expectedStartTime: "2013-16-22T05:10Z" + actualStartTime: "2013-16-22T05:05Z" + expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time) + actualEndTime: "2013-16-22T06:00Z" <-- (20 min late relative to expected end) + expectedDuration: 3600000 <-- (expected duration in milliseconds) + actualDuration: 3300000 <-- (actual duration in milliseconds) + notificationMessage: "My Job has encountered an SLA event!" + upstreamApps: "dependent-app-1, dependent-app-2" + +} +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Scenario_3:_Coordinator_Action_Duration_Miss"></a>Scenario 3: Coordinator Action Duration_Miss</h3> +<p><b>Request:</b></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=id=000001-1238791320234-oozie-joe-C +</pre></div></div> + +<p><b>JSON Response</b></p> + +<div> +<div> +<pre class="source">{ + + id : "000001-1238791320234-oozie-joe-C@2" + parentId : "000001-1238791320234-oozie-joe-C" + appType : "COORDINATOR_ACTION" + msgType : "SLA" + appName : "my-coord-app" + slaStatus : "MET" + jobStatus : "SUCCEEDED" + user: "joe" + nominalTime: "2013-16-22T05:00Z" + expectedStartTime: "2013-16-22T05:10Z" + actualStartTime: "2013-16-22T05:05Z" + expectedEndTime: "2013-16-22T05:40Z" + actualEndTime: "2013-16-22T05:30Z" + expectedDuration: 900000 <-- (expected duration in milliseconds) + actualDuration: 1500000 <- (actual duration in milliseconds) + notificationMessage: "My Job has encountered an SLA event!" + upstreamApps: "dependent-app-1, dependent-app-2" + +} +</pre></div></div> + +<p>Scenario #3 is particularly interesting because it is an overall “MET” because it met its expected End-time, but it is “Duration_Miss” because the actual run (between actual start and actual end) exceeded expected duration.</p></div> +<div class="section"> +<h3><a name="Scenario_4:_All_Coordinator_actions_in_a_Bundle"></a>Scenario 4: All Coordinator actions in a Bundle</h3> +<p><b>Request:</b></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=bundle=1234567-150130225116604-oozie-B;event_status=END_MISS +</pre></div></div> + +<p><b>JSON Response</b></p> + +<div> +<div> +<pre class="source">{ + id : "000001-1238791320234-oozie-joe-C@1" + parentId : "000001-1238791320234-oozie-joe-C" + appType : "COORDINATOR_ACTION" + msgType : "SLA" + appName : "my-coord-app" + slaStatus : "MET" + eventStatus : "START_MET,DURATION_MISS,END_MISS" + user: "joe" + nominalTime: "2014-01-10T12:00Z" + expectedStartTime: "2014-01-10T12:00Z" + actualStartTime: "2014-01-10T11:59Z" + startDelay: -1 + expectedEndTime: "2014-01-10T13:00Z" + actualEndTime: "2014-01-10T13:05Z" + endDelay: 5 + expectedDuration: 3600000 <-- (expected duration in milliseconds) + actualDuration: 3960000 <-- (actual duration in milliseconds) + durationDelay: 6 <-- (duration delay in minutes) +} +{ + id : "000001-1238791320234-oozie-joe-C@2" + parentId : "000001-1238791320234-oozie-joe-C" + appType : "COORDINATOR_ACTION" + msgType : "SLA" + appName : "my-coord-app" + slaStatus : "MET" + eventStatus : "START_MISS,DURATION_MET,END_MISS" + user: "joe" + nominalTime: "2014-01-11T12:00Z" + expectedStartTime: "2014-01-11T12:00Z" + actualStartTime: "2014-01-11T12:05Z" + startDelay: 5 + expectedEndTime: "2014-01-11T13:00Z" + actualEndTime: "2014-01-11T13:01Z" + endDelay: 1 + expectedDuration: 3600000 <-- (expected duration in milliseconds) + actualDuration: 3360000 <-- (actual duration in milliseconds) + durationDelay: -4 <-- (duration delay in minutes) +} +</pre></div></div> + +<p>Scenario #4 (All Coordinator actions in a Bundle) is to get SLA information of all coordinator actions under bundle job in one call. startDelay/durationDelay/endDelay values returned indicate how much delay compared to expected time (positive values in case of MISS, and negative values in case of MET).</p></div> +<div class="section"> +<h3><a name="Scenario_5:_Workflow_jobs_actually_started_in_a_24_hour_period"></a>Scenario 5: Workflow jobs actually started in a 24 hour period</h3> +<p><i>Request:</i></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=app_type=WORKFLOW_JOB;actualstart_after=2018-08-13T00:01Z;actualstart_before=2018-08-14T00:01Z +</pre></div></div> + +<p><i>JSON Response</i></p> + +<div> +<div> +<pre class="source">{ + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "one-op-wf", + "actualEnd": "Mon, 13 Aug 2018 14:49:21 GMT", + "actualDuration": 503, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "DURATION_MET,END_MISS,START_MISS", + "startDelay": 4531068, + "id": "0000001-180813160322492-oozie-test-W", + "lastModified": "Mon, 13 Aug 2018 14:49:31 GMT", + "user": "testuser", + "actualStart": "Mon, 13 Aug 2018 14:49:20 GMT", + "endDelay": 4531059 + }, + { + "nominalTime": "Fri, 01 Jan 2010 02:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 02:10:00 GMT", + "appName": "one-op-wf", + "actualEnd": "Mon, 13 Aug 2018 14:49:21 GMT", + "actualDuration": 222, + "expectedStart": "Fri, 01 Jan 2010 02:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "DURATION_MET,END_MISS,START_MISS", + "startDelay": 4531008, + "id": "0000002-180813160322492-oozie-test-W", + "lastModified": "Mon, 13 Aug 2018 14:49:41 GMT", + "user": "testuser", + "actualStart": "Mon, 13 Aug 2018 14:49:21 GMT", + "endDelay": 4530999 + } +</pre></div></div> + +<p>Scenario #5 is to get SLA information of all workflow jobs by filtering for the actual start date instead of the nominal start date.</p></div> +<div class="section"> +<h3><a name="Scenario_6:_Workflow_jobs_executed_much_faster_than_required"></a>Scenario 6: Workflow jobs executed much faster than required</h3> +<p><i>Request:</i></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=app_type=WORKFLOW_JOB;expected_duration_min=10000;actual_duration_max=1000 +</pre></div></div> + +<p><i>JSON Response</i></p> + +<div> +<div> +<pre class="source">{ + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "one-op-wf", + "actualEnd": "Mon, 13 Aug 2018 14:49:21 GMT", + "actualDuration": 503, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "DURATION_MET,END_MISS,START_MISS", + "startDelay": 4531068, + "id": "0000001-180813160322492-oozie-test-W", + "lastModified": "Mon, 13 Aug 2018 14:49:31 GMT", + "user": "testuser", + "actualStart": "Mon, 13 Aug 2018 14:49:20 GMT", + "endDelay": 4531059 + }, + { + "nominalTime": "Fri, 01 Jan 2010 02:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 02:10:00 GMT", + "appName": "one-op-wf", + "actualEnd": "Mon, 13 Aug 2018 14:49:21 GMT", + "actualDuration": 222, + "expectedStart": "Fri, 01 Jan 2010 02:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "DURATION_MET,END_MISS,START_MISS", + "startDelay": 4531008, + "id": "0000002-180813160322492-oozie-test-W", + "lastModified": "Mon, 13 Aug 2018 14:49:41 GMT", + "user": "testuser", + "actualStart": "Mon, 13 Aug 2018 14:49:21 GMT", + "endDelay": 4530999 + } +</pre></div></div> + +<p>Scenario #6 is to get SLA information of all workflow jobs where the expected duration was more than 10 seconds (10000ms), but the actual duration was less than a second (1000ms).</p></div> +<div class="section"> +<h3><a name="Scenario_7:_Coordinator_actions_with_START_MET_or_END_MET_event_status"></a>Scenario 7: Coordinator actions with START_MET or END_MET event status</h3> +<p><i>Request:</i></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=app_type=COORDINATOR_ACTION;event_status=START_MET,END_MET +</pre></div></div> + +<p><i>JSON Response</i></p> + +<div> +<div> +<pre class="source"> { + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "aggregator-coord", + "actualEnd": "Wed, 29 Aug 2018 10:29:59 GMT", + "actualDuration": 167, + "expectedStart": "Tue, 18 Feb 2200 11:41:46 GMT", + "expectedDuration": 60000, + "parentId": "0000006-180829120813646-oozie-test-C", + "durationDelay": 0, + "slaStatus": "MISS", + "appType": "COORDINATOR_ACTION", + "slaAlertStatus": "Disabled", + "eventStatus": "START_MET,DURATION_MET,END_MISS", + "startDelay": -95446151, + "id": "0000006-180829120813646-oozie-test-C@1", + "lastModified": "Wed, 29 Aug 2018 10:30:07 GMT", + "user": "testuser", + "actualStart": "Wed, 29 Aug 2018 10:29:59 GMT", + "endDelay": 4553839 + }, + { + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 05 Jan 2029 11:39:31 GMT", + "appName": "aggregator-coord", + "actualEnd": "Wed, 29 Aug 2018 10:15:48 GMT", + "actualDuration": 394, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 60000, + "parentId": "0000000-180829120813646-oozie-test-C", + "durationDelay": 0, + "slaStatus": "MET", + "appType": "COORDINATOR_ACTION", + "slaAlertStatus": "Disabled", + "eventStatus": "START_MISS,DURATION_MET,END_MET", + "startDelay": 4553834, + "id": "0000000-180829120813646-oozie-test-C@1", + "lastModified": "Wed, 29 Aug 2018 10:15:57 GMT", + "user": "testuser", + "actualStart": "Wed, 29 Aug 2018 10:15:48 GMT", + "endDelay": -5446163 + }, + { + "nominalTime": "Fri, 01 Jan 2010 02:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 02:10:00 GMT", + "appName": "aggregator-coord", + "actualEnd": "Wed, 29 Aug 2018 10:29:59 GMT", + "actualDuration": 172, + "expectedStart": "Tue, 18 Feb 2200 12:41:46 GMT", + "expectedDuration": 60000, + "parentId": "0000006-180829120813646-oozie-test-C", + "durationDelay": 0, + "slaStatus": "MISS", + "appType": "COORDINATOR_ACTION", + "slaAlertStatus": "Disabled", + "eventStatus": "START_MET,DURATION_MET,END_MISS", + "startDelay": -95446211, + "id": "0000006-180829120813646-oozie-test-C@2", + "lastModified": "Wed, 29 Aug 2018 10:30:17 GMT", + "user": "testuser", + "actualStart": "Wed, 29 Aug 2018 10:29:59 GMT", + "endDelay": 4553779 + }, + { + "nominalTime": "Fri, 01 Jan 2010 02:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 05 Jan 2029 12:39:31 GMT", + "appName": "aggregator-coord", + "actualEnd": "Wed, 29 Aug 2018 10:15:48 GMT", + "actualDuration": 208, + "expectedStart": "Fri, 01 Jan 2010 02:01:00 GMT", + "expectedDuration": 60000, + "parentId": "0000000-180829120813646-oozie-test-C", + "durationDelay": 0, + "slaStatus": "MET", + "appType": "COORDINATOR_ACTION", + "slaAlertStatus": "Disabled", + "eventStatus": "START_MISS,DURATION_MET,END_MET", + "startDelay": 4553774, + "id": "0000000-180829120813646-oozie-test-C@2", + "lastModified": "Wed, 29 Aug 2018 10:16:07 GMT", + "user": "testuser", + "actualStart": "Wed, 29 Aug 2018 10:15:48 GMT", + "endDelay": -5446223 + } +</pre></div></div> + +<p>Scenario #7 shows the possibility of filtering multiple event statuses. We list two comma separated statuses (START_MET,END_MET) and list coordinator actions with either START_MET or END_MET event status.</p></div> +<div class="section"> +<h3><a name="Scenario_8:_Not_yet_started_workflow_jobs_expected_to_start_before_a_specified_date."></a>Scenario 8: Not yet started workflow jobs expected to start before a specified date.</h3> +<p><i>Request:</i></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=app_type=WORKFLOW_JOB;sla_status=NOT_STARTED;expectedstart_before=2018-08-14T00:01Z +</pre></div></div> + +<p><i>JSON Response</i></p> + +<div> +<div> +<pre class="source"> { + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "PREP", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "one-op-wf", + "actualEnd": null, + "actualDuration": -1, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 300000, + "slaStatus": "NOT_STARTED", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "START_MISS,END_MISS", + "startDelay": 4561259, + "id": "0000031-180903152228376-oozie-test-W", + "lastModified": "Mon, 03 Sep 2018 14:00:50 GMT", + "user": "testuser", + "actualStart": null, + "endDelay": 4561250 + } +</pre></div></div> + +<p>Scenario #8 shows the possibility to list problematic jobs even before they start. It also shows the possibility to combine several filter fields.</p></div> +<div class="section"> +<h3><a name="Scenario_9:_Filtering_for_app_name_using__wildchar"></a>Scenario 9: Filtering for app_name using % wildchar</h3> +<p><i>Request:</i></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=app_name=appname-%25 +</pre></div></div> + +<p>Note that the filter is URL encoded, its decoded value is <tt>app_name=appname-%</tt></p> +<p><i>JSON Response</i></p> + +<div> +<div> +<pre class="source"> "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "appname-2", + "actualEnd": "Wed, 19 Sep 2018 15:02:48 GMT", + "actualDuration": 245, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "START_MISS,END_MISS,DURATION_MET", + "startDelay": 4584361, + "id": "0000003-180919170132414-oozie-test-W", + "lastModified": "Wed, 19 Sep 2018 15:02:56 GMT", + "user": "testuser", + "actualStart": "Wed, 19 Sep 2018 15:02:48 GMT", + "endDelay": 4584352 + }, + { + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "appname-1", + "actualEnd": "Wed, 19 Sep 2018 15:02:23 GMT", + "actualDuration": 378, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "START_MISS,END_MISS,DURATION_MET", + "startDelay": 4584361, + "id": "0000001-180919170132414-oozie-test-W", + "lastModified": "Wed, 19 Sep 2018 15:02:26 GMT", + "user": "testuser", + "actualStart": "Wed, 19 Sep 2018 15:02:23 GMT", + "endDelay": 4584352 + } +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Scenario_9:_Filtering_for_app_name_using__wildchar_and_sorting_the_order_by_the_application_name_in_descending_order"></a>Scenario 9: Filtering for app_name using % wildchar and sorting the order by the application name in descending order</h3> +<p><i>Request:</i></p> + +<div> +<div> +<pre class="source">GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=app_name=appname-%25&sortby=appName&order=desc +</pre></div></div> + +<p>Note that the filter is URL encoded, its decoded value is <tt>app_name=appname-%</tt></p> +<p><i>JSON Response</i></p> + +<div> +<div> +<pre class="source">{ + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "appname-2", + "actualEnd": "Wed, 19 Sep 2018 15:02:48 GMT", + "actualDuration": 245, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "START_MISS,END_MISS,DURATION_MET", + "startDelay": 4584361, + "id": "0000003-180919170132414-oozie-test-W", + "lastModified": "Wed, 19 Sep 2018 15:02:56 GMT", + "user": "testuser", + "actualStart": "Wed, 19 Sep 2018 15:02:48 GMT", + "endDelay": 4584352 + }, + { + "nominalTime": "Fri, 01 Jan 2010 01:00:00 GMT", + "jobStatus": "SUCCEEDED", + "expectedEnd": "Fri, 01 Jan 2010 01:10:00 GMT", + "appName": "appname-1", + "actualEnd": "Wed, 19 Sep 2018 15:02:23 GMT", + "actualDuration": 378, + "expectedStart": "Fri, 01 Jan 2010 01:01:00 GMT", + "expectedDuration": 300000, + "durationDelay": -4, + "slaStatus": "MISS", + "appType": "WORKFLOW_JOB", + "slaAlertStatus": "Enabled", + "eventStatus": "START_MISS,END_MISS,DURATION_MET", + "startDelay": 4584361, + "id": "0000001-180919170132414-oozie-test-W", + "lastModified": "Wed, 19 Sep 2018 15:02:26 GMT", + "user": "testuser", + "actualStart": "Wed, 19 Sep 2018 15:02:23 GMT", + "endDelay": 4584352 + } +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Sample_Email_Alert"></a>Sample Email Alert</h3> + +<div> +<div> +<pre class="source">Subject: OOZIE - SLA END_MISS (AppName=wf-sla-job, JobID=0000004-130610225200680-oozie-oozi-W) + + +Status: + SLA Status - END_MISS + Job Status - RUNNING + Notification Message - Missed SLA for Data Pipeline job +Job Details: + App Name - wf-sla-job + App Type - WORKFLOW_JOB + User - strat_ci + Job ID - 0000004-130610225200680-oozie-oozi-W + Job URL - http://host.domain.com:4080/oozie//?job=0000004-130610225200680-oozie-oozi-W + Parent Job ID - N/A + Parent Job URL - N/A + Upstream Apps - wf-sla-up-app +SLA Details: + Nominal Time - Mon Jun 10 23:33:00 UTC 2013 + Expected Start Time - Mon Jun 10 23:35:00 UTC 2013 + Actual Start Time - Mon Jun 10 23:34:04 UTC 2013 + Expected End Time - Mon Jun 10 23:38:00 UTC 2013 + Expected Duration (in mins) - 5 + Actual Duration (in mins) - -1 +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Changing_job_SLA_definition_and_alerting"></a>Changing job SLA definition and alerting</h3> +<p>Following are ways to enable/disable SLA alerts for coordinator actions.</p> +<div class="section"> +<h4><a name="a1._Specify_in_Bundle_XML_during_submission."></a>1. Specify in Bundle XML during submission.</h4> +<p>Following properties can be specified in bundle xml as properties for coordinator.</p> +<p><tt>oozie.sla.disable.alerts.older.than</tt> this property can be specified in hours, the SLA notification for coord actions will be disabled whose nominal is time older then this value. Default is 48 hours.</p> + +<div> +<div> +<pre class="source"><property> + <name>oozie.sla.disable.alerts.older.than</name> + <value>12</value> +</property> +</pre></div></div> + +<p><tt>oozie.sla.disable.alerts</tt> List of coord actions to be disabled. Value can be specified as list of coord actions or date range.</p> + +<div> +<div> +<pre class="source"><property> + <name>oozie.sla.disable.alerts</name> + <value>1,3-4,7-10</value> +</property> +</pre></div></div> + +<p>Will disable alert for coord actions 1,3,5,7,8,9,10</p> +<p><tt>oozie.sla.enable.alerts</tt> List of coord actions to be enabled. Value can be specified as list of coord actions or date range.</p> + +<div> +<div> +<pre class="source"><property> + <name>oozie.sla.enable.alerts</name> + <value>2009-01-01T01:00Z::2009-05-31T23:59Z</value> +</property> +</pre></div></div> + +<p>This will enable SLA alert for coord actions whose nominal time is in between (inclusive) 2009-01-01T01:00Z and 2009-05-31T23:59Z.</p> +<p>ALL keyword can be specified to specify all actions. Below property will disable SLA notifications for all coord actions.</p> + +<div> +<div> +<pre class="source"><property> + <name>oozie.sla.disable.alerts</name> + <value>ALL</value> +</property> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="a2._Specify_during_Coordinator_job_submission_or_update"></a>2. Specify during Coordinator job submission or update</h4> +<p>Above properties can be specified in job.properties in <a href="DG_CommandLineTool.html#Updating_coordinator_definition_and_properties">Coord job update command</a>, in <a href="DG_CommandLineTool.html#Submitting_a_Workflow_Coordinator_or_Bundle_Job">Coord job submit command</a> or in <a href="DG_CommandLineTool.html#Running_a_Workflow_Coordinator_or_Bundle_Job">Coord job run command</a></p></div> +<div class="section"> +<h4><a name="a3._Change_using_command_line"></a>3. Change using command line</h4> +<p>Refer <a href="DG_CommandLineTool.html#Changing_job_SLA_definition_and_alerting">Changing job SLA definition and alerting</a> for commandline usage.</p></div> +<div class="section"> +<h4><a name="a4._Change_using_REST_API"></a>4. Change using REST API</h4> +<p>Refer the REST API <a href="WebServicesAPI.html#Changing_job_SLA_definition_and_alerting">Changing job SLA definition and alerting</a>.</p></div></div></div> +<div class="section"> +<h2><a name="In-memory_SLA_entries_and_database_content"></a>In-memory SLA entries and database content</h2> +<p>There are special circumstances when the in-memory <tt>SLACalcStatus</tt> entries can exist without the workflow or coordinator job or action instances in database. For example:</p> +<ul> + +<li>SLA tracked database content may already have been deleted, and <tt>SLA_SUMMARY</tt> entry is not present anymore in database</li> +<li>SLA tracked database content and <tt>SLA_SUMMARY</tt> entry aren’t yet present in database</li> +</ul> +<p>By the time <tt>SLAService</tt> scheduled job will be running, SLA map contents are checked. When the <tt>SLA_SUMMARY</tt> entry for the in-memory SLA entry is missing, a counter is increased. When this counter reaches the server-wide preconfigured value <tt>oozie.sla.service.SLAService.maximum.retry.count</tt> (by default <tt>3</tt>), in-memory SLA entry will get purged.</p></div> +<div class="section"> +<h2><a name="Known_issues"></a>Known issues</h2> +<p>There are two known issues when you define SLA for a workflow action. * If there are decision nodes and SLA is defined for a workflow action not in the execution path because of the decision node, you will still get an SLA_MISS notification. * If you have dangling action nodes in your workflow definition and SLA is defined for it, you will still get an SLA_MISS notification.</p> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2019 + <a href="https://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
Added: oozie/site/trunk/content/resources/docs/5.2.0/DG_ShellActionExtension.html URL: http://svn.apache.org/viewvc/oozie/site/trunk/content/resources/docs/5.2.0/DG_ShellActionExtension.html?rev=1870914&view=auto ============================================================================== --- oozie/site/trunk/content/resources/docs/5.2.0/DG_ShellActionExtension.html (added) +++ oozie/site/trunk/content/resources/docs/5.2.0/DG_ShellActionExtension.html Fri Dec 6 08:59:57 2019 @@ -0,0 +1,583 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2019-12-05 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20191205" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org/"; class="externalLink" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.2.0"> + 5.2.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-12-05</li> + <li id="projectVersion" class="pull-right"> + Version: 5.2.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p><hr /> +<h1>Oozie Shell Action Extension</h1> +<ul> +<li><a href="#Shell_Action">Shell Action</a> +<ul> +<li><a href="#Shell_Action_Configuration">Shell Action Configuration</a></li> +<li><a href="#Shell_Action_Logging">Shell Action Logging</a></li> +<li><a href="#Shell_Action_Limitations">Shell Action Limitations</a></li></ul></li> +<li><a href="#Appendix_Shell_XML-Schema">Appendix, Shell XML-Schema</a> +<ul> +<li><a href="#AE.A_Appendix_A_Shell_XML-Schema">AE.A Appendix A, Shell XML-Schema</a> +<ul> +<li><a href="#Shell_Action_Schema_Version_1.0">Shell Action Schema Version 1.0</a></li> +<li><a href="#Shell_Action_Schema_Version_0.3">Shell Action Schema Version 0.3</a></li> +<li><a href="#Shell_Action_Schema_Version_0.2">Shell Action Schema Version 0.2</a></li> +<li><a href="#Shell_Action_Schema_Version_0.1">Shell Action Schema Version 0.1</a></li></ul></li></ul></li></ul> + +<p><a name="ShellAction"></a></p> +<div class="section"> +<h2><a name="Shell_Action"></a>Shell Action</h2> +<p>The <tt>shell</tt> action runs a Shell command.</p> +<p>The workflow job will wait until the Shell command completes before continuing to the next action.</p> +<p>To run the Shell job, you have to configure the <tt>shell</tt> action with the <tt>job-tracker</tt>, <tt>name-node</tt> and Shell <tt>exec</tt> elements as well as the necessary arguments and configuration.</p> +<p>A <tt>shell</tt> action can be configured to create or delete HDFS directories before starting the Shell job.</p> +<p>Shell <i>launcher</i> configuration can be specified with a file, using the <tt>job-xml</tt> element, and inline, using the <tt>configuration</tt> elements.</p> +<p>Oozie EL expressions can be used in the inline configuration. Property values specified in the <tt>configuration</tt> element override values specified in the <tt>job-xml</tt> file.</p> +<p>Note that YARN <tt>yarn.resourcemanager.address</tt> (<tt>resource-manager</tt>) and HDFS <tt>fs.default.name</tt> (<tt>name-node</tt>) properties must not be present in the inline configuration.</p> +<p>As with Hadoop <tt>map-reduce</tt> jobs, it is possible to add files and archives in order to make them available to the Shell job. Refer to the [WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job] section for more information about this feature.</p> +<p>The output (STDOUT) of the Shell job can be made available to the workflow job after the Shell job ends. This information could be used from within decision nodes. If the output of the Shell job is made available to the workflow job the shell command must follow the following requirements:</p> +<ul> + +<li>The format of the output must be a valid Java Properties file.</li> +<li>The size of the output must not exceed 2KB.</li> +</ul> +<p><b>Syntax:</b></p> + +<div> +<div> +<pre class="source"><workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0"> + ... + <action name="[NODE-NAME]"> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>[RESOURCE-MANAGER]</resource-manager> + <name-node>[NAME-NODE]</name-node> + <prepare> + <delete path="[PATH]"/> + ... + <mkdir path="[PATH]"/> + ... + </prepare> + <job-xml>[SHELL SETTINGS FILE]</job-xml> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + <exec>[SHELL-COMMAND]</exec> + <argument>[ARG-VALUE]</argument> + ... + <argument>[ARG-VALUE]</argument> + <env-var>[VAR1=VALUE1]</env-var> + ... + <env-var>[VARN=VALUEN]</env-var> + <file>[FILE-PATH]</file> + ... + <archive>[FILE-PATH]</archive> + ... + <capture-output/> + </shell> + <ok to="[NODE-NAME]"/> + <error to="[NODE-NAME]"/> + </action> + ... +</workflow-app> +</pre></div></div> + +<p>The <tt>prepare</tt> element, if present, indicates a list of paths to delete or create before starting the job. Specified paths must start with <tt>hdfs://HOST:PORT</tt>.</p> +<p>The <tt>job-xml</tt> element, if present, specifies a file containing configuration for the Shell job. As of schema 0.2, multiple <tt>job-xml</tt> elements are allowed in order to specify multiple <tt>job.xml</tt> files.</p> +<p>The <tt>configuration</tt> element, if present, contains configuration properties that are passed to the Shell job.</p> +<p>The <tt>exec</tt> element must contain the path of the Shell command to execute. The arguments of Shell command can then be specified using one or more <tt>argument</tt> element.</p> +<p>The <tt>argument</tt> element, if present, contains argument to be passed to the Shell command.</p> +<p>The <tt>env-var</tt> element, if present, contains the environment to be passed to the Shell command. <tt>env-var</tt> should contain only one pair of environment variable and value. If the pair contains the variable such as $PATH, it should follow the Unix convention such as PATH=$PATH:mypath. Don’t use ${PATH} which will be substituted by Oozie’s EL evaluator.</p> +<p>A <tt>shell</tt> action creates a Hadoop configuration. The Hadoop configuration is made available as a local file to the Shell application in its running directory. The exact file path is exposed to the spawned shell using the environment variable called <tt>OOZIE_ACTION_CONF_XML</tt>.The Shell application can access the environment variable to read the action configuration XML file path.</p> +<p>If the <tt>capture-output</tt> element is present, it indicates Oozie to capture output of the STDOUT of the shell command execution. The Shell command output must be in Java Properties file format and it must not exceed 2KB. From within the workflow definition, the output of an Shell action node is accessible via the <tt>String action:output(String node, String key)</tt> function (Refer to section ‘4.2.6 Action EL Functions’).</p> +<p>All the above elements can be parameterized (templatized) using EL expressions.</p> +<p><b>Example:</b></p> +<p>How to run any shell script or perl script or CPP executable</p> + +<div> +<div> +<pre class="source"><workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'> + <start to='shell1' /> + <action name='shell1'> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> + <name-node>${nameNode}</name-node> + <configuration> + <property> + <name>mapred.job.queue.name</name> + <value>${queueName}</value> + </property> + </configuration> + <exec>${EXEC}</exec> + <argument>A</argument> + <argument>B</argument> + <file>${EXEC}#${EXEC}</file> <!--Copy the executable to compute node's current working directory --> + </shell> + <ok to="end" /> + <error to="fail" /> + </action> + <kill name="fail"> + <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> + </kill> + <end name='end' /> +</workflow-app> +</pre></div></div> + +<p>The corresponding job properties file used to submit Oozie job could be as follows:</p> + +<div> +<div> +<pre class="source">oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script + +#Execute is expected to be in the Workflow directory. +#Shell Script to run +EXEC=script.sh +#CPP executable. Executable should be binary compatible to the compute node OS. +#EXEC=hello +#Perl script +#EXEC=script.pl + +resourceManager=localhost:8032 +nameNode=hdfs://localhost:8020 +queueName=default + +</pre></div></div> + +<p>How to run any java program bundles in a jar.</p> + +<div> +<div> +<pre class="source"><workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'> + <start to='shell1' /> + <action name='shell1'> + <shell xmlns="uri:oozie:shell-action:1.0"> + <resource-manager>${resourceManager}</resource-manager> + <name-node>${nameNode}</name-node> + <configuration> + <property> + <name>mapred.job.queue.name</name> + <value>${queueName}</value> + </property> + </configuration> + <exec>java</exec> + <argument>-classpath</argument> + <argument>./${EXEC}:$CLASSPATH</argument> + <argument>Hello</argument> + <file>${EXEC}#${EXEC}</file> <!--Copy the jar to compute node current working directory --> + </shell> + <ok to="end" /> + <error to="fail" /> + </action> + <kill name="fail"> + <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> + </kill> + <end name='end' /> +</workflow-app> +</pre></div></div> + +<p>The corresponding job properties file used to submit Oozie job could be as follows:</p> + +<div> +<div> +<pre class="source">oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script + +#Hello.jar file is expected to be in the Workflow directory. +EXEC=Hello.jar + +resourceManager=localhost:8032 +nameNode=hdfs://localhost:8020 +queueName=default +</pre></div></div> + +<div class="section"> +<h3><a name="Shell_Action_Configuration"></a>Shell Action Configuration</h3> +<ul> + +<li><tt>oozie.action.shell.setup.hadoop.conf.dir</tt> - Generates a config directory with various core/hdfs/yarn/mapred-site.xml files and points <tt>HADOOP_CONF_DIR</tt> and <tt>YARN_CONF_DIR</tt> env-vars to it, before the Script is invoked. XML is sourced from the action configuration. Useful when the Shell script passed uses various <tt>hadoop</tt> commands. Default is false.</li> +<li><tt>oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties</tt> - When <tt>oozie.action.shell.setup.hadoop.conf.dir</tt> is enabled, toggle if a log4j.properties file should also be written under the configuration files directory. Default is true.</li> +<li><tt>oozie.action.shell.setup.hadoop.conf.dir.log4j.content</tt> - When <tt>oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties</tt> is enabled, the content to write into the log4j.properties file under the configuration files directory. Default is a simple console based stderr logger, as presented below:</li> +</ul> + +<div> +<div> +<pre class="source">log4j.rootLogger=${hadoop.root.logger} +hadoop.root.logger=INFO,console +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.target=System.err +log4j.appender.console.layout=org.apache.log4j.PatternLayout +log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Shell_Action_Logging"></a>Shell Action Logging</h3> +<p>Shell action’s stdout and stderr output are redirected to the Oozie Launcher map-reduce job task STDOUT that runs the shell command.</p> +<p>From Oozie web-console, from the Shell action pop up using the ‘Console URL’ link, it is possible to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.</p></div> +<div class="section"> +<h3><a name="Shell_Action_Limitations"></a>Shell Action Limitations</h3> +<p>Although Shell action can execute any shell command, there are some limitations.</p> +<ul> + +<li>No interactive command is supported.</li> +<li>Command can’t be executed as different user using sudo.</li> +<li>User has to explicitly upload the required 3rd party packages (such as jar, so lib, executable etc). Oozie provides a way using <file> and <archive> tag through Hadoop’s Distributed Cache to upload.</li> +<li>Since Oozie will execute the shell command into a Hadoop compute node, the default installation of utility in the compute node might not be fixed. However, the most common unix utilities are usually installed on all compute nodes. It is important to note that Oozie could only support the commands that are installed into the compute nodes or that are uploaded through Distributed Cache.</li> +</ul></div></div> +<div class="section"> +<h2><a name="Appendix_Shell_XML-Schema"></a>Appendix, Shell XML-Schema</h2> +<div class="section"> +<h3><a name="AE.A_Appendix_A_Shell_XML-Schema"></a>AE.A Appendix A, Shell XML-Schema</h3> +<div class="section"> +<h4><a name="Shell_Action_Schema_Version_1.0"></a>Shell Action Schema Version 1.0</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:shell="uri:oozie:shell-action:1.0" + elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:1.0"> +. + <xs:include schemaLocation="oozie-common-1.0.xsd"/> +. + <xs:element name="shell" type="shell:ACTION"/> +. + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:choice> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:choice> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="launcher" type="shell:LAUNCHER" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> +. +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Shell_Action_Schema_Version_0.3"></a>Shell Action Schema Version 0.3</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:shell="uri:oozie:shell-action:0.3" elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:0.3"> + + <xs:element name="shell" type="shell:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="FLAG"/> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Shell_Action_Schema_Version_0.2"></a>Shell Action Schema Version 0.2</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:shell="uri:oozie:shell-action:0.2" elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:0.2"> + + <xs:element name="shell" type="shell:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="FLAG"/> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Shell_Action_Schema_Version_0.1"></a>Shell Action Schema Version 0.1</h4> + +<div> +<div> +<pre class="source"><?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" + xmlns:shell="uri:oozie:shell-action:0.1" elementFormDefault="qualified" + targetNamespace="uri:oozie:shell-action:0.1"> + + <xs:element name="shell" type="shell:ACTION"/> + + <xs:complexType name="ACTION"> + <xs:sequence> + <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/> + <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/> + <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="FLAG"/> + + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="PREPARE"> + <xs:sequence> + <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/> + <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="DELETE"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="MKDIR"> + <xs:attribute name="path" type="xs:string" use="required"/> + </xs:complexType> + +</xs:schema> +</pre></div></div> + +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2019 + <a href="https://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}