Added: websites/staging/oozie/trunk/content/docs/5.1.0/AG_Monitoring.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.1.0/AG_Monitoring.html (added) +++ websites/staging/oozie/trunk/content/docs/5.1.0/AG_Monitoring.html Wed Dec 19 15:50:03 2018 @@ -0,0 +1,414 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Oozie Monitoring</h1> +<ul> +<li><a href="#Oozie_Instrumentation">Oozie Instrumentation</a> +<ul> +<li><a href="#Variables">Variables</a></li> +<li><a href="#Samplers_-_Poll_data_at_a_fixed_interval_default_1_sec_and_report_an_average_utilization_over_a_longer_period_of_time_default_60_seconds.">Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).</a></li> +<li><a href="#Counters_-_Maintain_statistics_about_the_number_of_times_an_event_has_occurred_for_the_running_Oozie_instance._The_values_are_reset_if_the_Oozie_instance_is_restarted.">Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.</a></li> +<li><a href="#Timers_-_Maintain_information_about_the_time_spent_in_various_operations.">Timers - Maintain information about the time spent in various operations.</a></li></ul></li> +<li><a href="#Oozie_JVM_Thread_Dump">Oozie JVM Thread Dump</a></li> +<li><a href="#Monitoring_Database_Schema_Integrity">Monitoring Database Schema Integrity</a></li></ul> + +<div class="section"> +<h2><a name="Oozie_Instrumentation"></a>Oozie Instrumentation</h2> +<p>Oozie code is instrumented in several places to collect runtime metrics. The instrumentation data can be used to determine the health of the system, performance of the system, and to tune the system.</p> +<p>This comes in two flavors:</p> +<ul> + +<li>metrics (by default enabled since 5.0.0)</li> +<li>instrumentation (deprecated and by default disabled since 5.0.0)</li> +</ul> +<p>The instrumentation is accessible via the Admin web-services API (see the <a href="WebServicesAPI.html#Oozie_Metrics">metrics</a> and <a href="WebServicesAPI.html#Oozie_Instrumentation">instrumentation</a> Web Services API documentations for more details) and is also written on regular intervals to an instrumentation log.</p> +<p>Instrumentation data includes variables, samplers, timers and counters.</p> +<div class="section"> +<h3><a name="Variables"></a>Variables</h3> +<ul> + +<li> + +<p>oozie</p> +<ul> + +<li>version: Oozie build version.</li> +</ul> +</li> +<li> + +<p>configuration</p> +<ul> + +<li>config.dir: directory from where the configuration files are loaded. If null, all configuration files are loaded from the classpath. <a href="AG_Install.html#Oozie_Configuration">Configuration files are described here</a>.</li> +<li>config.file: the Oozie custom configuration for the instance.</li> +</ul> +</li> +<li> + +<p>jvm</p> +<ul> + +<li>free.memory</li> +<li>max.memory</li> +<li>total.memory</li> +</ul> +</li> +<li> + +<p>locks</p> +<ul> + +<li>locks: Locks are used by Oozie to synchronize access to workflow and action entries when the database being used does not support ‘select for update’ queries. (MySQL supports ‘select for update’).</li> +</ul> +</li> +<li> + +<p>logging</p> +<ul> + +<li>config.file: Log4j ‘.properties’ configuration file.</li> +<li>from.classpath: whether the config file has been read from the classpath or from the config directory.</li> +<li>reload.interval: interval at which the config file will be reloaded. 0 if the config file will never be reloaded, when loaded from the classpath is never reloaded.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="Samplers_-_Poll_data_at_a_fixed_interval_default_1_sec_and_report_an_average_utilization_over_a_longer_period_of_time_default_60_seconds."></a>Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).</h3> +<p>Poll for data over fixed interval and generate an average over the time interval. Unless specified, all samplers in Oozie work on a 1 minute interval.</p> +<ul> + +<li> + +<p>callablequeue</p> +<ul> + +<li>delayed.queue.size: The size of the delayed command queue.</li> +<li>queue.size: The size of the command queue.</li> +<li>threads.active: The number of threads processing callables.</li> +</ul> +</li> +<li> + +<p>jdbc:</p> +<ul> + +<li>connections.active: Active Connections over the past minute.</li> +</ul> +</li> +<li> + +<p>webservices: Requests to the Oozie HTTP endpoints over the last minute.</p> +<ul> + +<li>admin</li> +<li>callback</li> +<li>job</li> +<li>jobs</li> +<li>requests</li> +<li>version</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="Counters_-_Maintain_statistics_about_the_number_of_times_an_event_has_occurred_for_the_running_Oozie_instance._The_values_are_reset_if_the_Oozie_instance_is_restarted."></a>Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.</h3> +<ul> + +<li>action.executors - Counters related to actions. +<ul> + +<li>[action_type]#action.[operation_performed] (start, end, check, kill)</li> +<li>[action_type]#ex.[exception_type] (transient, non-transient, error, failed)</li> +<li>e.g.</li> +</ul> +</li> +</ul> + +<div> +<div> +<pre class="source">ssh#action.end: 306 +ssh#action.start: 316 +</pre></div></div> + +<ul> + +<li> + +<p>callablequeue - count of events in various execution queues.</p> +<ul> + +<li>delayed.queued: Number of commands queued with a delay.</li> +<li>executed: Number of executions from the queue.</li> +<li>failed: Number of queue attempts which failed.</li> +<li>queued: Number of queued commands.</li> +</ul> +</li> +<li> + +<p>commands: Execution Counts for various commands. This data is generated for all commands.</p> +<ul> + +<li>action.end</li> +<li>action.notification</li> +<li>action.start</li> +<li>callback</li> +<li>job.info</li> +<li>job.notification</li> +<li>purge</li> +<li>signal</li> +<li>start</li> +<li>submit</li> +</ul> +</li> +<li> + +<p>jobs: Job Statistics</p> +<ul> + +<li>start: Number of started jobs.</li> +<li>submit: Number of submitted jobs.</li> +<li>succeeded: Number of jobs which succeeded.</li> +<li>kill: Number of killed jobs.</li> +</ul> +</li> +<li> + +<p>authorization</p> +<ul> + +<li>failed: Number of failed authorization attempts.</li> +</ul> +</li> +<li> + +<p>webservices: Number of request to various web services along with the request type.</p> +<ul> + +<li>failed: total number of failed requests.</li> +<li>requests: total number of requests.</li> +<li>admin</li> +<li>admin-GET</li> +<li>callback</li> +<li>callback-GET</li> +<li>jobs</li> +<li>jobs-GET</li> +<li>jobs-POST</li> +<li>version</li> +<li>version-GET</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="Timers_-_Maintain_information_about_the_time_spent_in_various_operations."></a>Timers - Maintain information about the time spent in various operations.</h3> +<ul> + +<li> + +<p>action.executors - Counters related to actions.</p> +<ul> + +<li>[action_type]#action.[operation_performed] (start, end, check, kill)</li> +</ul> +</li> +<li> + +<p>callablequeue</p> +<ul> + +<li>time.in.queue: Time a callable spent in the queue before being processed.</li> +</ul> +</li> +<li> + +<p>commands: Generated for all Commands.</p> +<ul> + +<li>action.end</li> +<li>action.notification</li> +<li>action.start</li> +<li>callback</li> +<li>job.info</li> +<li>job.notification</li> +<li>purge</li> +<li>signal</li> +<li>start</li> +<li>submit</li> +</ul> +</li> +<li> + +<p>db - Timers related to various database operations.</p> +<ul> + +<li>create-workflow</li> +<li>load-action</li> +<li>load-pending-actions</li> +<li>load-running-actions</li> +<li>load-workflow</li> +<li>load-workflows</li> +<li>purge-old-workflows</li> +<li>save-action</li> +<li>update-action</li> +<li>update-workflow</li> +</ul> +</li> +<li> + +<p>webservices</p> +<ul> + +<li>admin</li> +<li>admin-GET</li> +<li>callback</li> +<li>callback-GET</li> +<li>jobs</li> +<li>jobs-GET</li> +<li>jobs-POST</li> +<li>version</li> +<li>version-GET</li> +</ul> +</li> +</ul></div></div> +<div class="section"> +<h2><a name="Oozie_JVM_Thread_Dump"></a>Oozie JVM Thread Dump</h2> +<p>The <tt>admin/jvminfo.jsp</tt> servlet can be used to get some basic jvm stats and thread dump. For eg: <tt>http://localhost:11000/oozie/admin/jvminfo.jsp?cpuwatch=1000&threadsort=cpu</tt>. It takes the following optional query parameters:</p> +<ul> + +<li>threadsort - The order in which the threads are sorted for display. Valid values are name, cpu, state. Default is state.</li> +<li>cpuwatch - Time interval in milliseconds to monitor cpu usage of threads. Default value is 0.</li> +</ul></div> +<div class="section"> +<h2><a name="Monitoring_Database_Schema_Integrity"></a>Monitoring Database Schema Integrity</h2> +<p>Oozie stores all of its state in a database. Hence, ensuring that the database schema is correct is very important to ensuring that Oozie is healthy and behaves correctly. To help with this, Oozie includes a <tt>SchemaCheckerService</tt> which periodically runs and performs a series of checks on the database schema. More specifically, it checks the following:</p> +<ul> + +<li>Existence of the required tables</li> +<li>Existence of the required columns in each table</li> +<li>Each column has the correct type and default value</li> +<li>Existence of the required primary keys and indexes</li> +</ul> +<p>After each run, the <tt>SchemaCheckerService</tt> writes the result of the checks to the Oozie log and to the “schema-checker.status” instrumentation variable. If there’s a problem, it will be logged at the ERROR level, while correct checks are logged at the DEBUG level.</p> +<p>By default, the <tt>SchemaCheckerService</tt> runs every 7 days. This can be configured by <tt>oozie.service.SchemaCheckerService.check.interval</tt></p> +<p>By default, the <tt>SchemaCheckerService</tt> will consider “extra” tables, columns, and indexes to be incorrect. Advanced users who have added additional tables, columns, and indexes can tell Oozie to ignore these by setting <tt>oozie.service.SchemaCheckerService.ignore.extras</tt> to <tt>false</tt>.</p> +<p>The <tt>SchemaCheckerService</tt> currently only supports MySQL, PostgreSQL, and Oracle databases. SQL Server and Derby are currently not supported.</p> +<p>When Oozie HA is enabled, only one of the Oozie servers will perform the checks.</p> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
Added: websites/staging/oozie/trunk/content/docs/5.1.0/AG_OozieLogging.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.1.0/AG_OozieLogging.html (added) +++ websites/staging/oozie/trunk/content/docs/5.1.0/AG_OozieLogging.html Wed Dec 19 15:50:03 2018 @@ -0,0 +1,208 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Oozie Logging</h1> +<ul> +<li><a href="#Default_Oozie_Logging">Default Oozie Logging</a> +<ul> +<li><a href="#Default_Configuration">Default Configuration</a></li> +<li><a href="#Restrictions">Restrictions</a></li></ul></li> +<li><a href="#Previous_Default_Oozie_Logging">Previous Default Oozie Logging</a> +<ul> +<li><a href="#Previous_Default_Configuration">Previous Default Configuration</a></li> +<li><a href="#Restrictions">Restrictions</a></li></ul></li> +<li><a href="#Other_Oozie_Logging">Other Oozie Logging</a></li></ul> + +<div class="section"> +<h2><a name="Default_Oozie_Logging"></a>Default Oozie Logging</h2> +<p>Oozie’s logging properties can be configured in its log4j properties file (default is <tt>oozie-log4j.properties</tt>). Most log messages are configured by default to be written to the <tt>oozie</tt> appender.</p> +<p>The default configuration for the <tt>oozie</tt> appender is shown below.</p> +<div class="section"> +<h3><a name="Default_Configuration"></a>Default Configuration</h3> + +<div> +<div> +<pre class="source">log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppender +log4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy +log4j.appender.oozie.File=${oozie.log.dir}/oozie.log +log4j.appender.oozie.Append=true +log4j.appender.oozie.layout=org.apache.log4j.PatternLayout +log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n +log4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH} +log4j.appender.oozie.RollingPolicy.MaxHistory=720 +</pre></div></div> + +<p>In this configuration, the active log file will be named <tt>oozie.log</tt> and all old log files will be named <tt>oozie.log-yyyy-MM-dd-HH</tt> (where <tt>yyyy-MM-dd-HH</tt> is the time that that log file was created; e.g. 2012-07-21-05). All log files are in the same directory (whatever <tt>oozie.log.dir</tt> is assigned to). A maximum of 720 older log files will be retained. The active log file is rolled every hour, so 720 old logs means that they are kept for 30 days before being deleted.</p> +<p>To keep all old logs instead of deleting them, <tt>log4j.appender.oozie.RollingPolicy.MaxHistory</tt> can be set to <tt>-1</tt>. Additionally, <tt>log4j.appender.oozie.RollingPolicy</tt> can be set to <tt>org.apache.log4j.rolling.TimeBasedRollingPolicy</tt>, which has the same exact behavior as <tt>org.apache.oozie.util.OozieRollingPolicy</tt> except that it does not delete old logs.</p></div> +<div class="section"> +<h3><a name="Restrictions"></a>Restrictions</h3> +<p>In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in the <tt>oozie-log4j.properties</tt> file):</p> +<ul> + +<li> + +<p>The appender that Oozie uses must be named “oozie” (i.e. <tt>log4j.appender.oozie</tt>)</p> +</li> +<li> + +<p><tt>log4j.appender.oozie.RollingPolicy.FileNamePattern</tt> must end with “-%d{yyyy-MM-dd-HH}.gz” or “-%d{yyyy-MM-dd-HH}”. If it ends with “.gz” the old logs will be compressed when rolled</p> +</li> +<li> + +<p><tt>log4j.appender.oozie.RollingPolicy.FileNamePattern</tt> must start with the value of <tt>log4j.appender.oozie.File</tt></p> +</li> +</ul></div></div> +<div class="section"> +<h2><a name="Previous_Default_Oozie_Logging"></a>Previous Default Oozie Logging</h2> +<p>Oozie previously used the logging configuration shown below as the default for the <tt>oozie</tt> appender. The other appender that Oozie writes to still use a configuration similar to this.</p> +<div class="section"> +<h3><a name="Previous_Default_Configuration"></a>Previous Default Configuration</h3> + +<div> +<div> +<pre class="source">log4j.appender.oozie=org.apache.log4j.DailyRollingFileAppender +log4j.appender.oozie.File=${oozie.log.dir}/oozie.log +log4j.appender.oozie.Append=true +log4j.appender.oozie.layout=org.apache.log4j.PatternLayout +log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n +log4j.appender.oozie.DatePattern='.'yyyy-MM-dd-HH +</pre></div></div> + +<p>In this configuration, the active log file will be named <tt>oozie.log</tt> and all old log files will be named <tt>oozie.log.yyyy-MM-dd-HH</tt> (where <tt>yyyy-MM-dd-HH</tt> is the time that the log file was created; e.g. 2012-07-21-05). All log files are in the same directory (whatever <tt>oozie.log.dir</tt> is assigned to). All older log files are retained. The active log file is rolled every hour.</p></div> +<div class="section"> +<h3><a name="Restrictions"></a>Restrictions</h3> +<p>In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in the <tt>oozie-log4j.properties</tt> file):</p> +<ul> + +<li> + +<p>The appender that Oozie uses must be named “oozie” (i.e. <tt>log4j.appender.oozie</tt>)</p> +</li> +<li> + +<p><tt>log4j.appender.oozie.DatePattern</tt> must end with either “dd” or “HH”. If it ends with “HH”, the log will be rolled every hour; if it ends with “dd”, the log will be rolled every day.</p> +</li> +</ul></div></div> +<div class="section"> +<h2><a name="Other_Oozie_Logging"></a>Other Oozie Logging</h2> +<p>While Oozie can technically use any valid log4j Appender or configurations that violate the above restrictions, certain features related to logs may be disabled and/or not work correctly, and is thus not advised.</p> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/oozie/trunk/content/docs/5.1.0/AG_OozieUpgrade.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.1.0/AG_OozieUpgrade.html (added) +++ websites/staging/oozie/trunk/content/docs/5.1.0/AG_OozieUpgrade.html Wed Dec 19 15:50:03 2018 @@ -0,0 +1,181 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p> +<h1>Oozie Upgrade</h1> +<ul> +<li><a href="#Preparation">Preparation</a></li> +<li><a href="#Oozie_Server_Upgrade">Oozie Server Upgrade</a></li> +<li><a href="#Oozie_Client_Upgrade">Oozie Client Upgrade</a></li></ul> + +<div class="section"> +<h2><a name="Preparation"></a>Preparation</h2> +<p>Make sure there are not Workflows in RUNNING or SUSPENDED status, otherwise the database upgrade will fail.</p> +<p>Shutdown Oozie and backup the Oozie database.</p> +<p>Copy the oozie-site.xml from your current setup.</p></div> +<div class="section"> +<h2><a name="Oozie_Server_Upgrade"></a>Oozie Server Upgrade</h2> +<p>Expand the new Oozie tarball in a new location.</p> +<p>Edit the new <tt>oozie-site.xml</tt> setting all custom properties values from the old <tt>oozie-site.xml</tt></p> +<p>IMPORTANT: From Oozie 2.x to Oozie 3.x the names of the database configuration properties have changed. Their prefix has changed from <tt>oozie.service.StoreService.*</tt> to <tt>oozie.service.JPAService.*</tt>. Make sure you are using the new prefix.</p> +<p>After upgrading the Oozie server, the <tt>oozie-setup.sh</tt> MUST be rerun before starting the upgraded Oozie server.</p> +<p>Oozie database migration is required when there Oozie database schema changes, like upgrading from Oozie 2.x to Oozie 3.x.</p> +<p>Configure the oozie-site.xml with the correct database configuration properties as explained in the ‘Database Configuration’ section in <a href="AG_Install.html">Oozie Install</a>.</p> +<p>Once <tt>oozie-site.xml</tt> has been configured with the database configuration execute the <tt>ooziedb.sh</tt> command line tool to upgrade the database:</p> + +<div> +<div> +<pre class="source">$ bin/ooziedb.sh upgrade -run + +Validate DB Connection. +DONE +Check DB schema exists +DONE +Check OOZIE_SYS table does not exist +DONE +Verify there are not active Workflow Jobs +DONE +Create SQL schema +DONE +DONE +Create OOZIE_SYS table +DONE +Upgrade COORD_JOBS new columns default values. +DONE +Upgrade COORD_JOBS & COORD_ACTIONS status values. +DONE +Table 'WF_ACTIONS' column 'execution_path', length changed to 1024 +DONE + +Oozie DB has been upgraded to Oozie version '3.2.0' + +The SQL commands have been written to: /tmp/ooziedb-5737263881793872034.sql + +$ +</pre></div></div> + +<p>The new version of the Oozie server is ready to be started.</p> +<p>NOTE: If using MySQL or Oracle, copy the corresponding JDBC driver JAR file to the <tt>libext/</tt> directory before running the <tt>ooziedb.sh</tt> command line tool.</p> +<p>NOTE: If instead using the ‘-run’ option, the <tt>-sqlfile <FILE></tt> option is used, then all the database changes will be written to the specified file and the database won’t be modified.</p></div> +<div class="section"> +<h2><a name="Oozie_Client_Upgrade"></a>Oozie Client Upgrade</h2> +<p>While older Oozie clients work with newer Oozie server, to have access to all the functionality of the Oozie server the same version of Oozie client should be installed and used by users.</p> +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/oozie/trunk/content/docs/5.1.0/BundleFunctionalSpec.html ============================================================================== --- websites/staging/oozie/trunk/content/docs/5.1.0/BundleFunctionalSpec.html (added) +++ websites/staging/oozie/trunk/content/docs/5.1.0/BundleFunctionalSpec.html Wed Dec 19 15:50:03 2018 @@ -0,0 +1,530 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-12-19 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181219" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Oozie – </title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="https://oozie.apache.org/"; id="bannerLeft"> + <img src="https://oozie.apache.org/images/oozie_200x.png"; alt="Oozie"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../" title="Apache"> + Apache</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../../" title="Oozie"> + Oozie</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="../" title="docs"> + docs</a> + <span class="divider">/</span> + </li> + <li class=""> + <a href="./" title="5.1.0"> + 5.1.0</a> + <span class="divider">/</span> + </li> + <li class="active "></li> + + + + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-19</li> + <li id="projectVersion" class="pull-right"> + Version: 5.1.0 + </li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <p><a href="index.html">::Go back to Oozie Documentation Index::</a></p><hr /> +<h1>Oozie Bundle Specification</h1> +<p>The goal of this document is to define a new oozie abstraction called bundle system specialized in submitting and maintaining a set of coordinator applications.</p> +<ul> +<li><a href="#Changelog">Changelog</a></li> +<li><a href="#a1._Bundle_Overview">1. Bundle Overview</a></li> +<li><a href="#a2._Definitions">2. Definitions</a></li> +<li><a href="#a3._Expression_Language_for_Parameterization">3. Expression Language for Parameterization</a></li> +<li><a href="#a4._Bundle_Job">4. Bundle Job</a> +<ul> +<li><a href="#a4.1._Bundle_Job_Status">4.1. Bundle Job Status</a></li> +<li><a href="#a4.2._Transitions_of_Bundle_Job_Status">4.2. Transitions of Bundle Job Status</a></li> +<li><a href="#a4.3._Details_of_Status_Transitions">4.3. Details of Status Transitions</a></li> +<li><a href="#a4.3.__Bundle_Application_Definition">4.3. Bundle Application Definition</a></li> +<li><a href="#a4.4.__Bundle_Formal_Parameters">4.4. Bundle Formal Parameters</a></li></ul></li> +<li><a href="#a5._User_Propagation">5. User Propagation</a></li> +<li><a href="#a6._Bundle_Application_Deployment">6. Bundle Application Deployment</a> +<ul> +<li><a href="#a6.1._Organizing_Bundle_Applications">6.1. Organizing Bundle Applications</a></li></ul></li> +<li><a href="#a7._Bundle_Job_Submission">7. Bundle Job Submission</a></li> +<li><a href="#a8._Bundle_Rerun">8. Bundle Rerun</a> +<ul> +<li><a href="#a8.1_Rerunning_a_Bundle_Job">8.1 Rerunning a Bundle Job</a></li> +<li><a href="#a8.2_Rerun_Arguments">8.2 Rerun Arguments</a></li></ul></li> +<li><a href="#Appendixes">Appendixes</a> +<ul> +<li><a href="#Appendix_A_Oozie_Bundle_XML-Schema">Appendix A, Oozie Bundle XML-Schema</a> +<ul> +<li><a href="#Oozie_Bundle_Schema_0.1">Oozie Bundle Schema 0.1</a></li> +<li><a href="#Oozie_Bundle_Schema_0.2">Oozie Bundle Schema 0.2</a></li></ul></li></ul></li></ul> + +<div class="section"> +<h2><a name="Changelog"></a>Changelog</h2></div> +<div class="section"> +<h2><a name="a1._Bundle_Overview"></a>1. Bundle Overview</h2> +<p>Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun in the bundle level resulting a better and easy operational control.</p> +<p>More specifically, the oozie <b>Bundle</b> system allows the user to define and execute a bunch of coordinator applications often called a data pipeline. There is no explicit dependency among the coordinator applications in a bundle. However, a user could use the data dependency of coordinator applications to create an implicit data application pipeline.</p></div> +<div class="section"> +<h2><a name="a2._Definitions"></a>2. Definitions</h2> +<p><b>Kick-off-time:</b> The time when a bundle should start and submit coordinator applications.</p> +<p><b>Bundle Application:</b> A bundle application defines a set of coordinator applications and when to start those. Normally, bundle applications are parameterized. A bundle application is written in XML.</p> +<p><b>Bundle Job:</b> A bundle job is an executable instance of a bundle application. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.</p> +<p><b>Bundle Definition Language:</b> The language used to describe bundle applications.</p></div> +<div class="section"> +<h2><a name="a3._Expression_Language_for_Parameterization"></a>3. Expression Language for Parameterization</h2> +<p>Bundle application definitions can be parameterized with variables.</p> +<p>At job submission time all the parameters are resolved into concrete values.</p> +<p>The parameterization of bundle definitions is done using JSP Expression Language syntax from the <a class="externalLink" href="http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html";>JSP 2.0 Specification (JSP.2.3)</a>, allowing not only to support variables as parameters but also complex expressions.</p> +<p>EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.</p></div> +<div class="section"> +<h2><a name="a4._Bundle_Job"></a>4. Bundle Job</h2> +<div class="section"> +<h3><a name="a4.1._Bundle_Job_Status"></a>4.1. Bundle Job Status</h3> +<p>At any time, a bundle job is in one of the following status: <b>PREP, RUNNING, RUNNINGWITHERROR, SUSPENDED, PREPSUSPENDED, SUSPENDEDWITHERROR, PAUSED, PAUSEDWITHERROR, PREPPAUSED, SUCCEEDED, DONEWITHERROR, KILLED, FAILED</b>.</p></div> +<div class="section"> +<h3><a name="a4.2._Transitions_of_Bundle_Job_Status"></a>4.2. Transitions of Bundle Job Status</h3> +<p>Valid bundle job status transitions are:</p> +<ul> + +<li><b>PREP –> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED</b></li> +<li><b>RUNNING –> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED</b></li> +<li><b>RUNNINGWITHERROR –> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | FAILED | KILLED</b></li> +<li><b>PREPSUSPENDED –> PREP | KILLED</b></li> +<li><b>SUSPENDED –> RUNNING | KILLED</b></li> +<li><b>SUSPENDEDWITHERROR –> RUNNINGWITHERROR | KILLED</b></li> +<li><b>PREPPAUSED –> PREP | KILLED</b></li> +<li><b>PAUSED –> SUSPENDED | RUNNING | KILLED</b></li> +<li><b>PAUSEDWITHERROR –> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED</b></li> +</ul></div> +<div class="section"> +<h3><a name="a4.3._Details_of_Status_Transitions"></a>4.3. Details of Status Transitions</h3> +<p>When a bundle job is submitted, oozie parses the bundle job XML. Oozie then creates a record for the bundle with status <b>PREP</b> and returns a unique ID.</p> +<p>When a user requests to suspend a bundle job that is in <b>PREP</b> state, oozie puts the job in status <b>PREPSUSPENDED</b>. Similarly, when pause time reaches for a bundle job with <b>PREP</b> status, oozie puts the job in status <b>PREPPAUSED</b>.</p> +<p>Conversely, when a user requests to resume a <b>PREPSUSPENDED</b> bundle job, oozie puts the job in status <b>PREP</b>. And when pause time is reset for a bundle job that is in <b>PREPPAUSED</b> state, oozie puts the job in status <b>PREP</b>.</p> +<p>There are two ways a bundle job could be started.</p> +<ul> + +<li> + +<p>If <tt>kick-off-time</tt> (defined in the bundle xml) reaches. The default value is null which means starts coordinators NOW.</p> +</li> +<li> + +<p>If user sends a start request to START the bundle.</p> +</li> +</ul> +<p>When a bundle job starts, oozie puts the job in status <b>RUNNING</b> and it submits all the coordinator jobs. If any coordinator job goes to <b>FAILED/KILLED/DONEWITHERROR</b> state, the bundle job is put in <b>RUNNINGWITHERROR</b></p> +<p>When a user requests to kill a bundle job, oozie puts the job in status <b>KILLED</b> and it sends kill to all submitted coordinator jobs.</p> +<p>When a user requests to suspend a bundle job that is in <b>RUNNING</b> status, oozie puts the job in status <b>SUSPENDED</b> and it suspends all submitted coordinator jobs. Similarly, when a user requests to suspend a bundle job that is in <b>RUNNINGWITHERROR</b> status, oozie puts the job in status <b>SUSPENDEDWITHERROR</b> and it suspends all submitted coordinator jobs.</p> +<p>When pause time reaches for a bundle job that is in <b>RUNNING</b> status, oozie puts the job in status <b>PAUSED</b>. When pause time reaches for a bundle job that is in <b>RUNNINGWITHERROR</b> status, oozie puts the job in status <b>PAUSEDWITHERROR</b>.</p> +<p>Conversely, when a user requests to resume a <b>SUSPENDED</b> bundle job, oozie puts the job in status <b>RUNNING</b>. Similarly, when a user requests to resume a <b>SUSPENDEDWITHERROR</b> bundle job, oozie puts the job in status <b>RUNNINGWITHERROR</b>. And when pause time is reset for a bundle job and job status is <b>PAUSED</b>, oozie puts the job in status <b>RUNNING</b>. Similarly, when the pause time is reset for a bundle job and job status is <b>PAUSEDWITHERROR</b>, oozie puts the job in status <b>RUNNINGWITHERROR</b></p> +<p>When all the coordinator jobs finish, oozie updates the bundle status accordingly. If all coordinators reaches to the <i>same</i> terminal state, bundle job status also move to the same status. For example, if all coordinators are <b>SUCCEEDED</b>, oozie puts the bundle job into <b>SUCCEEDED</b> status. However, if all coordinator jobs don’t finish with the same status, oozie puts the bundle job into <b>DONEWITHERROR</b>.</p></div> +<div class="section"> +<h3><a name="a4.3.__Bundle_Application_Definition"></a>4.3. Bundle Application Definition</h3> +<p>A bundle definition is defined in XML by a name, controls and one or more coordinator application specifications:</p> +<ul> + +<li><b><font color="#0000ff"> name: </font></b> The name for the bundle job.</li> +<li><b><font color="#0000ff"> controls: </font></b> The control specification for the bundle. +<ul> + +<li><b><font color="#0000ff"> kick-off-time: </font></b> It defines when the bundle job should start and submit the coordinator applications. This field is optional and the default is <b>NOW</b> that means the job should start right-a-way.</li> +</ul> +</li> +<li><b><font color="#0000ff"> coordinator: </font></b> Coordinator application specification. There should be at least one coordinator application in any bundle. +<ul> + +<li><b><font color="#0000ff"> name: </font></b> Name of the coordinator application. It can be used for referring this application through bundle to control such as kill, suspend, rerun.</li> +<li><b><font color="#0000ff"> enabled: </font></b> Enabled can be used to enable or disable a coordinator. It is optional. The default value for enabled is true.</li> +<li><b><font color="#0000ff"> app-path: </font></b> Path of the coordinator application definition in hdfs. This is a mandatory element.</li> +<li><b><font color="#0000ff"> configuration: </font></b> A hadoop like configuration to parameterize corresponding coordinator application. This is optional.</li> +<li><b><font color="#0000ff"> Parameterization: </font></b> Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as <tt>${NAME}</tt> variables within the bundle application definition. Configuration properties that are not a valid Java identifier, for example <tt>job.tracker</tt>, are available via the <tt>${bundle:conf(String name)}</tt> function. Valid Java identifier properties are available via this function as well.</li> +</ul> +</li> +</ul> +<p><b><font color="#800080">Syntax: </font></b></p> + +<div> +<div> +<pre class="source"> <bundle-app name=[NAME] xmlns='uri:oozie:bundle:0.1'> + <controls> + <kick-off-time>[DATETIME]</kick-off-time> + </controls> + <coordinator name=[NAME] enabled=[TRUE | FALSE] > + <app-path>[COORD-APPLICATION-PATH]</app-path> + <configuration> + <property> + <name>[PROPERTY-NAME]</name> + <value>[PROPERTY-VALUE]</value> + </property> + ... + </configuration> + </coordinator> + ... +</bundle-app> +</pre></div></div> + +<p><b><font color="#008000"> Examples: </font></b></p> +<p><b>A Bundle Job that maintains two coordinator applications:</b></p> + +<div> +<div> +<pre class="source"><bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.1'> + <controls> + <kick-off-time>${kickOffTime}</kick-off-time> + </controls> + <coordinator name="${bundle:conf('coordName1')}" > + <app-path>${appPath}</app-path> + <configuration> + <property> + <name>startTime1</name> + <value>${bundle:conf('coord1.startTime1')}</value> + </property> + <property> + <name>endTime1</name> + <value>${END_TIME}</value> + </property> + </configuration> + </coordinator> + <coordinator name='coordJobFromBundle2' > + <app-path>${appPath2}</app-path> + <configuration> + <property> + <name>startTime2</name> + <value>${START_TIME2}</value> + </property> + <property> + <name>endTime2</name> + <value>${END_TIME2}</value> + </property> + </configuration> + </coordinator> +</bundle-app> +</pre></div></div> +</div> +<div class="section"> +<h3><a name="a4.4.__Bundle_Formal_Parameters"></a>4.4. Bundle Formal Parameters</h3> +<p>As of schema 0.2, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.</p> +<p><b>Example:</b></p> +<p>The previous Bundle Job application definition with formal parameters:</p> + +<div> +<div> +<pre class="source"><bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.2'> + <parameters> + <property> + <name>appPath</name> + </property> + <property> + <name>appPath2</name> + <value>hdfs://foo:8020/user/joe/job/job.properties</value> + </property> + </parameters> + <controls> + <kick-off-time>${kickOffTime}</kick-off-time> + </controls> + <coordinator name='coordJobFromBundle1' > + <app-path>${appPath}</app-path> + <configuration> + <property> + <name>startTime1</name> + <value>${START_TIME}</value> + </property> + <property> + <name>endTime1</name> + <value>${END_TIME}</value> + </property> + </configuration> + </coordinator> + <coordinator name='coordJobFromBundle2' > + <app-path>${appPath2}</app-path> + <configuration> + <property> + <name>startTime2</name> + <value>${START_TIME2}</value> + </property> + <property> + <name>endTime2</name> + <value>${END_TIME2}</value> + </property> + </configuration> + </coordinator> +</bundle-app> +</pre></div></div> + +<p>In the above example, if <tt>appPath</tt> is not specified, Oozie will print an error message instead of submitting the job. If <tt>appPath2</tt> is not specified, Oozie will use the default value, <tt>hdfs://foo:8020/user/joe/job/job.properties</tt>.</p></div></div> +<div class="section"> +<h2><a name="a5._User_Propagation"></a>5. User Propagation</h2> +<p>When submitting a bundle job, the configuration must contain a <tt>user.name</tt> property. If security is enabled, Oozie must ensure that the value of the <tt>user.name</tt> property in the configuration match the user credentials present in the protocol (web services) request.</p> +<p>When submitting a bundle job, the configuration may contain the <tt>oozie.job.acl</tt> property (the <tt>group.name</tt> property has been deprecated). If authorization is enabled, this property is treated as as the ACL for the job, it can contain user and group IDs separated by commas.</p> +<p>The specified user and ACL are assigned to the created bundle job.</p> +<p>Oozie must propagate the specified user and ACL to the system executing its children jobs (coordinator jobs).</p></div> +<div class="section"> +<h2><a name="a6._Bundle_Application_Deployment"></a>6. Bundle Application Deployment</h2> +<p>A bundle application consist exclusively of bundle application definition and associated coordinator application specifications. They must be installed in an HDFS directory. To submit a job for a bundle application, the full HDFS path to bundle application definition must be specified.</p> +<div class="section"> +<h3><a name="a6.1._Organizing_Bundle_Applications"></a>6.1. Organizing Bundle Applications</h3> +<p>TBD.</p></div></div> +<div class="section"> +<h2><a name="a7._Bundle_Job_Submission"></a>7. Bundle Job Submission</h2> +<p>When a bundle job is submitted to Oozie, the submitter must specified all the required job properties plus the HDFS path to the bundle application definition for the job.</p> +<p>The bundle application definition HDFS path must be specified in the ‘oozie.bundle.application.path’ job property.</p> +<p>All the bundle job properties, the HDFS path for the bundle application, the ‘user.name’ and ‘oozie.job.acl’ must be submitted to the Oozie using an XML configuration file (Hadoop XML configuration file).</p> +<p><b><font color="#008000"> Example: </font></b>:</p> + +<div> +<div> +<pre class="source"><?xml version="1.0" encoding="UTF-8"?> +<configuration> + <property> + <name>user.name</name> + <value>joe</value> + </property> + <property> + <name>oozie.bundle.application.path</name> + <value>hdfs://foo:8020/user/joe/mybundles/hello-bundle1.xml</value> + </property> + ... +</configuration> +</pre></div></div> +</div> +<div class="section"> +<h2><a name="a8._Bundle_Rerun"></a>8. Bundle Rerun</h2> +<div class="section"> +<h3><a name="a8.1_Rerunning_a_Bundle_Job"></a>8.1 Rerunning a Bundle Job</h3> +<p>Oozie provides a way of rerunning a bundle job. The user could request to rerun a subset of coordinators within a bundle by defining a list of coordinator’s names. In addition, a user could define a list of dates or ranges of dates (in UTC format) to rerun for those time windows. There is a way of asking whether to cleanup all output directories before rerun. By default, oozie will remove all output directories. Moreover, there is an option by which a user could ask to re-calculate the dynamic input directories defined by latest function in coordinators.</p></div> +<div class="section"> +<h3><a name="a8.2_Rerun_Arguments"></a>8.2 Rerun Arguments</h3> + +<div> +<div> +<pre class="source">$oozie job -rerun <bundle_Job_id> [-coordinator <list of coordinator name separate by comma> +[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] + [-nocleanup] [-refresh] +</pre></div></div> + +<ul> + +<li>The <tt>rerun</tt> option reruns a bundle job that is <i>not</i> in (<tt>KILLED</tt>, <tt>FAILED</tt>, <tt>PREP</tt>, <tt>PREPPAUSED</tt>, <tt>PREPSUSPENDED</tt>).</li> +<li>Rerun a bundle job that is in <tt>PAUSED</tt> state will reset the paused time.</li> +<li>The option -coordinator determines the name of coordinator that will be rerun. By default all coordinators are rerun.</li> +<li>Multiple ranges can be used in -date. See the above examples.</li> +<li>The dates specified in -date must be UTC.</li> +<li>If -nocleanup is given, corresponding coordinator directories will not be removed; otherwise the ‘output-event’ will be deleted.</li> +<li>If -refresh is set, new dataset is re-evaluated for latest() and future() for the corresponding coordinators.</li> +<li>If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked for the corresponding coordinators.</li> +</ul> +<p>After the command is executed the rerun bundle job will be in <tt>RUNNING</tt> status.</p> +<p>Refer to the <a href="DG_CoordinatorRerun.html">Rerunning Coordinator Actions</a> for details on rerun of coordinator job.</p></div></div> +<div class="section"> +<h2><a name="Appendixes"></a>Appendixes</h2> +<div class="section"> +<h3><a name="Appendix_A_Oozie_Bundle_XML-Schema"></a>Appendix A, Oozie Bundle XML-Schema</h3> +<div class="section"> +<h4><a name="Oozie_Bundle_Schema_0.1"></a>Oozie Bundle Schema 0.1</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.1" + elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.1"> + + <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/> + <xs:simpleType name="IDENTIFIER"> + <xs:restriction base="xs:string"> + <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39})"/> + </xs:restriction> + </xs:simpleType> + <xs:complexType name="BUNDLE-APP"> + <xs:sequence> + <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/> + <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/> + </xs:sequence> + <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/> + </xs:complexType> + <xs:complexType name="CONTROLS"> + <xs:sequence minOccurs="0" maxOccurs="1"> + <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + <xs:complexType name="COORDINATOR"> + <xs:sequence minOccurs="1" maxOccurs="1"> + <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/> + <xs:attribute name="critical" type="xs:string" use="optional"/> + </xs:complexType> + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +</xs:schema> +</pre></div></div> +</div> +<div class="section"> +<h4><a name="Oozie_Bundle_Schema_0.2"></a>Oozie Bundle Schema 0.2</h4> + +<div> +<div> +<pre class="source"><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.2" + elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.2"> + + <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/> + <xs:simpleType name="IDENTIFIER"> + <xs:restriction base="xs:string"> + <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}"/> + </xs:restriction> + </xs:simpleType> + <xs:complexType name="BUNDLE-APP"> + <xs:sequence> + <xs:element name="parameters" type="bundle:PARAMETERS" minOccurs="0" maxOccurs="1"/> + <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/> + <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/> + </xs:sequence> + <xs:attribute name="name" type="xs:string" use="required"/> + </xs:complexType> + <xs:complexType name="PARAMETERS"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="0" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + <xs:complexType name="CONTROLS"> + <xs:sequence minOccurs="0" maxOccurs="1"> + <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + </xs:complexType> + <xs:complexType name="COORDINATOR"> + <xs:sequence minOccurs="1" maxOccurs="1"> + <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/> + <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/> + </xs:sequence> + <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/> + <xs:attribute name="critical" type="xs:string" use="optional"/> + <xs:attribute name="enabled" type="xs:string" use="optional"/> + </xs:complexType> + <xs:complexType name="CONFIGURATION"> + <xs:sequence> + <xs:element name="property" minOccurs="1" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/> + <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> +</xs:schema> +</pre></div></div> + +<p><a href="index.html">::Go back to Oozie Documentation Index::</a></p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + <p >Copyright © 2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All rights reserved. + + </p> + </div> + + + </div> + </footer> + </body> +</html>
![https://oozie.apache.org/images/oozie_200x.png"](https://oozie.apache.org/images/oozie_200x.png"){kind=link}