Added: aurora/site/publish/documentation/0.12.0/configuration-tutorial/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.12.0/configuration-tutorial/index.html?rev=1733548&view=auto ============================================================================== --- aurora/site/publish/documentation/0.12.0/configuration-tutorial/index.html (added) +++ aurora/site/publish/documentation/0.12.0/configuration-tutorial/index.html Fri Mar 4 02:43:01 2016 @@ -0,0 +1,1098 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/configuration-tutorial/'" + value="0.12.0"> + <option value="0.12.0" + selected="selected"> + 0.12.0 + (latest) + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="aurora-configuration-tutorial">Aurora Configuration Tutorial</h1> + +<p>How to write Aurora configuration files, including feature descriptions +and best practices. When writing a configuration file, make use of +<code>aurora job inspect</code>. It takes the same job key and configuration file +arguments as <code>aurora job create</code> or <code>aurora update start</code>. It first ensures the +configuration parses, then outputs it in human-readable form.</p> + +<p>You should read this after going through the general <a href="/documentation/0.12.0/tutorial/">Aurora Tutorial</a>.</p> + +<ul> +<li><a href="#aurora-configuration-tutorial">Aurora Configuration Tutorial</a> + +<ul> +<li><a href="#the-basics">The Basics</a> + +<ul> +<li><a href="#use-bottom-to-top-object-ordering">Use Bottom-To-Top Object Ordering</a></li> +</ul></li> +<li><a href="#an-example-configuration-file">An Example Configuration File</a></li> +<li><a href="#defining-process-objects">Defining Process Objects</a></li> +<li><a href="#getting-your-code-into-the-sandbox">Getting Your Code Into The Sandbox</a></li> +<li><a href="#defining-task-objects">Defining Task Objects</a> + +<ul> +<li><a href="#sequentialtask-running-processes-in-parallel-or-sequentially">SequentialTask: Running Processes in Parallel or Sequentially</a></li> +<li><a href="#simpletask">SimpleTask</a></li> +<li><a href="#combining-tasks">Combining tasks</a></li> +</ul></li> +<li><a href="#defining-job-objects">Defining Job Objects</a></li> +<li><a href="#the-jobs-list">The jobs List</a></li> +<li><a href="#templating">Templating</a> + +<ul> +<li><a href="#templating-1-binding-in-pystachio">Templating 1: Binding in Pystachio</a></li> +<li><a href="#structurals-in-pystachio--aurora">Structurals in Pystachio / Aurora</a> + +<ul> +<li><a href="#mustaches-within-structurals">Mustaches Within Structurals</a></li> +</ul></li> +<li><a href="#templating-2-structurals-are-factories">Templating 2: Structurals Are Factories</a> + +<ul> +<li><a href="#a-second-way-of-templating">A Second Way of Templating</a></li> +</ul></li> +<li><a href="#advanced-binding">Advanced Binding</a> + +<ul> +<li><a href="#bind-syntax">Bind Syntax</a></li> +<li><a href="#binding-complex-objects">Binding Complex Objects</a> + +<ul> +<li><a href="#lists"></a></li> +<li><a href="#maps"></a></li> +<li><a href="#structurals"></a></li> +</ul></li> +</ul></li> +<li><a href="#structural-binding">Structural Binding</a></li> +</ul></li> +<li><a href="#configuration-file-writing-tips-and-best-practices">Configuration File Writing Tips And Best Practices</a> + +<ul> +<li><a href="#use-as-few-aurora-files-as-possible">Use As Few .aurora Files As Possible</a></li> +<li><a href="#avoid-boilerplate">Avoid Boilerplate</a></li> +<li><a href="#thermos-uses-bash-but-thermos-is-not-bash">Thermos Uses bash, But Thermos Is Not bash</a> + +<ul> +<li><a href="#bad">Bad</a></li> +<li><a href="#good">Good</a></li> +</ul></li> +<li><a href="#rarely-use-functions-in-your-configurations">Rarely Use Functions In Your Configurations</a> + +<ul> +<li><a href="#bad-1">Bad</a></li> +<li><a href="#good-1">Good</a></li> +</ul></li> +</ul></li> +</ul></li> +</ul> + +<h2 id="the-basics">The Basics</h2> + +<p>To run a job on Aurora, you must specify a configuration file that tells +Aurora what it needs to know to schedule the job, what Mesos needs to +run the tasks the job is made up of, and what Thermos needs to run the +processes that make up the tasks. This file must have +a<code>.aurora</code> suffix.</p> + +<p>A configuration file defines a collection of objects, along with parameter +values for their attributes. An Aurora configuration file contains the +following three types of objects:</p> + +<ul> +<li>Job</li> +<li>Task</li> +<li>Process</li> +</ul> + +<p>A configuration also specifies a list of <code>Job</code> objects assigned +to the variable <code>jobs</code>.</p> + +<ul> +<li>jobs (list of defined Jobs to run)</li> +</ul> + +<p>The <code>.aurora</code> file format is just Python. However, <code>Job</code>, <code>Task</code>, +<code>Process</code>, and other classes are defined by a type-checked dictionary +templating library called <em>Pystachio</em>, a powerful tool for +configuration specification and reuse. Pystachio objects are tailored +via {{}} surrounded templates.</p> + +<p>When writing your <code>.aurora</code> file, you may use any Pystachio datatypes, as +well as any objects shown in the <a href="/documentation/0.12.0/configuration-reference/"><em>Aurora+Thermos Configuration +Reference</em></a>, without <code>import</code> statements - the +Aurora config loader injects them automatically. Other than that, an <code>.aurora</code> +file works like any other Python script.</p> + +<p><a href="/documentation/0.12.0/configuration-reference/"><em>Aurora+Thermos Configuration Reference</em></a> +has a full reference of all Aurora/Thermos defined Pystachio objects.</p> + +<h3 id="use-bottom-to-top-object-ordering">Use Bottom-To-Top Object Ordering</h3> + +<p>A well-structured configuration starts with structural templates (if +any). Structural templates encapsulate in their attributes all the +differences between Jobs in the configuration that are not directly +manipulated at the <code>Job</code> level, but typically at the <code>Process</code> or <code>Task</code> +level. For example, if certain processes are invoked with slightly +different settings or input.</p> + +<p>After structural templates, define, in order, <code>Process</code>es, <code>Task</code>s, and +<code>Job</code>s.</p> + +<p>Structural template names should be <em>UpperCamelCased</em> and their +instantiations are typically <em>UPPER_SNAKE_CASED</em>. <code>Process</code>, <code>Task</code>, +and <code>Job</code> names are typically <em>lower_snake_cased</em>. Indentation is typically 2 +spaces.</p> + +<h2 id="an-example-configuration-file">An Example Configuration File</h2> + +<p>The following is a typical configuration file. Don’t worry if there are +parts you don’t understand yet, but you may want to refer back to this +as you read about its individual parts. Note that names surrounded by +curly braces {{}} are template variables, which the system replaces with +bound values for the variables.</p> +<pre class="highlight plaintext"><code># --- templates here --- +class Profile(Struct): + package_version = Default(String, 'live') + java_binary = Default(String, '/usr/lib/jvm/java-1.7.0-openjdk/bin/java') + extra_jvm_options = Default(String, '') + parent_environment = Default(String, 'prod') + parent_serverset = Default(String, + '/foocorp/service/bird/{{parent_environment}}/bird') + +# --- processes here --- +main = Process( + name = 'application', + cmdline = '{{profile.java_binary}} -server -Xmx1792m ' + '{{profile.extra_jvm_options}} ' + '-jar application.jar ' + '-upstreamService {{profile.parent_serverset}}' +) + +# --- tasks --- +base_task = SequentialTask( + name = 'application', + processes = [ + Process( + name = 'fetch', + cmdline = 'curl -O + https://packages.foocorp.com/{{profile.package_version}}/application.jar'), + ] +) + + # not always necessary but often useful to have separate task + # resource classes + staging_task = base_task(resources = + Resources(cpu = 1.0, + ram = 2048*MB, + disk = 1*GB)) +production_task = base_task(resources = + Resources(cpu = 4.0, + ram = 2560*MB, + disk = 10*GB)) + +# --- job template --- +job_template = Job( + name = 'application', + role = 'myteam', + contact = '[email protected]', + instances = 20, + service = True, + task = production_task +) + +# -- profile instantiations (if any) --- +PRODUCTION = Profile() +STAGING = Profile( + extra_jvm_options = '-Xloggc:gc.log', + parent_environment = 'staging' +) + +# -- job instantiations -- +jobs = [ + job_template(cluster = 'cluster1', environment = 'prod') + .bind(profile = PRODUCTION), + + job_template(cluster = 'cluster2', environment = 'prod') + .bind(profile = PRODUCTION), + + job_template(cluster = 'cluster1', + environment = 'staging', + service = False, + task = staging_task, + instances = 2) + .bind(profile = STAGING), +] +</code></pre> + +<h2 id="defining-process-objects">Defining Process Objects</h2> + +<p>Processes are handled by the Thermos system. A process is a single +executable step run as a part of an Aurora task, which consists of a +bash-executable statement.</p> + +<p>The key (and required) <code>Process</code> attributes are:</p> + +<ul> +<li> <code>name</code>: Any string which is a valid Unix filename (no slashes, +NULLs, or leading periods). The <code>name</code> value must be unique relative +to other Processes in a <code>Task</code>.</li> +<li> <code>cmdline</code>: A command line run in a bash subshell, so you can use +bash scripts. Nothing is supplied for command-line arguments, +so <code>$*</code> is unspecified.</li> +</ul> + +<p>Many tiny processes make managing configurations more difficult. For +example, the following is a bad way to define processes.</p> +<pre class="highlight plaintext"><code>copy = Process( + name = 'copy', + cmdline = 'curl -O https://packages.foocorp.com/app.zip' +) +unpack = Process( + name = 'unpack', + cmdline = 'unzip app.zip' +) +remove = Process( + name = 'remove', + cmdline = 'rm -f app.zip' +) +run = Process( + name = 'app', + cmdline = 'java -jar app.jar' +) +run_task = Task( + processes = [copy, unpack, remove, run], + constraints = order(copy, unpack, remove, run) +) +</code></pre> + +<p>Since <code>cmdline</code> runs in a bash subshell, you can chain commands +with <code>&&</code> or <code>||</code>.</p> + +<p>When defining a <code>Task</code> that is just a list of Processes run in a +particular order, use <code>SequentialTask</code>, as described in the <a href="#Task"><em>Defining</em> +<code>Task</code> <em>Objects</em></a> section. The following simplifies and combines the +above multiple <code>Process</code> definitions into just two.</p> +<pre class="highlight plaintext"><code>stage = Process( + name = 'stage', + cmdline = 'curl -O https://packages.foocorp.com/app.zip && ' + 'unzip app.zip && rm -f app.zip') + +run = Process(name = 'app', cmdline = 'java -jar app.jar') + +run_task = SequentialTask(processes = [stage, run]) +</code></pre> + +<p><code>Process</code> also has optional attributes to customize its behaviour. Details can be found in the <a href="/documentation/0.12.0/configuration-reference/#process-objects"><em>Aurora+Thermos Configuration Reference</em></a>.</p> + +<h2 id="getting-your-code-into-the-sandbox">Getting Your Code Into The Sandbox</h2> + +<p>When using Aurora, you need to get your executable code into its “sandbox”, specifically +the Task sandbox where the code executes for the Processes that make up that Task.</p> + +<p>Each Task has a sandbox created when the Task starts and garbage +collected when it finishes. All of a Task’s processes run in its +sandbox, so processes can share state by using a shared current +working directory.</p> + +<p>Typically, you save this code somewhere. You then need to define a Process +in your <code>.aurora</code> configuration file that fetches the code from that somewhere +to where the slave can see it. For a public cloud, that can be anywhere public on +the Internet, such as S3. For a private cloud internal storage, you need to put in +on an accessible HDFS cluster or similar storage.</p> + +<p>The template for this Process is:</p> +<pre class="highlight plaintext"><code><name> = Process( + name = '<name>' + cmdline = '<command to copy and extract code archive into current working directory>' +) +</code></pre> + +<p>Note: Be sure the extracted code archive has an executable.</p> + +<h2 id="defining-task-objects">Defining Task Objects</h2> + +<p>Tasks are handled by Mesos. A task is a collection of processes that +runs in a shared sandbox. It’s the fundamental unit Aurora uses to +schedule the datacenter; essentially what Aurora does is find places +in the cluster to run tasks.</p> + +<p>The key (and required) parts of a Task are:</p> + +<ul> +<li><p><code>name</code>: A string giving the Task’s name. By default, if a Task is +not given a name, it inherits the first name in its Process list.</p></li> +<li><p><code>processes</code>: An unordered list of Process objects bound to the Task. +The value of the optional <code>constraints</code> attribute affects the +contents as a whole. Currently, the only constraint, <code>order</code>, determines if +the processes run in parallel or sequentially.</p></li> +<li><p><code>resources</code>: A <code>Resource</code> object defining the Task’s resource + footprint. A <code>Resource</code> object has three attributes: + - <code>cpu</code>: A Float, the fractional number of cores the Task + requires. + - <code>ram</code>: An Integer, RAM bytes the Task requires. + - <code>disk</code>: An integer, disk bytes the Task requires.</p></li> +</ul> + +<p>A basic Task definition looks like:</p> +<pre class="highlight plaintext"><code>Task( + name="hello_world", + processes=[Process(name = "hello_world", cmdline = "echo hello world")], + resources=Resources(cpu = 1.0, + ram = 1*GB, + disk = 1*GB)) +</code></pre> + +<p>A Task has optional attributes to customize its behaviour. Details can be found in the <a href="/documentation/0.12.0/configuration-reference/#task-object"><em>Aurora+Thermos Configuration Reference</em></a></p> + +<h3 id="sequentialtask-running-processes-in-parallel-or-sequentially">SequentialTask: Running Processes in Parallel or Sequentially</h3> + +<p>By default, a Task with several Processes runs them in parallel. There +are two ways to run Processes sequentially:</p> + +<ul> +<li><p>Include an <code>order</code> constraint in the Task definition’s <code>constraints</code> +attribute whose arguments specify the processes’ run order:</p> +<pre class="highlight plaintext"><code>Task( ... processes=[process1, process2, process3], + constraints = order(process1, process2, process3), ...) +</code></pre></li> +<li><p>Use <code>SequentialTask</code> instead of <code>Task</code>; it automatically runs +processes in the order specified in the <code>processes</code> attribute. No +<code>constraint</code> parameter is needed:</p> +<pre class="highlight plaintext"><code>SequentialTask( ... processes=[process1, process2, process3] ...) +</code></pre></li> +</ul> + +<h3 id="simpletask">SimpleTask</h3> + +<p>For quickly creating simple tasks, use the <code>SimpleTask</code> helper. It +creates a basic task from a provided name and command line using a +default set of resources. For example, in a .<code>aurora</code> configuration +file:</p> +<pre class="highlight plaintext"><code>SimpleTask(name="hello_world", command="echo hello world") +</code></pre> + +<p>is equivalent to</p> +<pre class="highlight plaintext"><code>Task(name="hello_world", + processes=[Process(name = "hello_world", cmdline = "echo hello world")], + resources=Resources(cpu = 1.0, + ram = 1*GB, + disk = 1*GB)) +</code></pre> + +<p>The simplest idiomatic Job configuration thus becomes:</p> +<pre class="highlight plaintext"><code>import os +hello_world_job = Job( + task=SimpleTask(name="hello_world", command="echo hello world"), + role=os.getenv('USER'), + cluster="cluster1") +</code></pre> + +<p>When written to <code>hello_world.aurora</code>, you invoke it with a simple +<code>aurora job create cluster1/$USER/test/hello_world hello_world.aurora</code>.</p> + +<h3 id="combining-tasks">Combining tasks</h3> + +<p><code>Tasks.concat</code>(synonym,<code>concat_tasks</code>) and +<code>Tasks.combine</code>(synonym,<code>combine_tasks</code>) merge multiple Task definitions +into a single Task. It may be easier to define complex Jobs +as smaller constituent Tasks. But since a Job only includes a single +Task, the subtasks must be combined before using them in a Job. +Smaller Tasks can also be reused between Jobs, instead of having to +repeat their definition for multiple Jobs.</p> + +<p>With both methods, the merged Task takes the first Task’s name. The +difference between the two is the result Task’s process ordering.</p> + +<ul> +<li><p><code>Tasks.combine</code> runs its subtasks’ processes in no particular order. +The new Task’s resource consumption is the sum of all its subtasks’ +consumption.</p></li> +<li><p><code>Tasks.concat</code> runs its subtasks in the order supplied, with each +subtask’s processes run serially between tasks. It is analogous to +the <code>order</code> constraint helper, except at the Task level instead of +the Process level. The new Task’s resource consumption is the +maximum value specified by any subtask for each Resource attribute +(cpu, ram and disk).</p></li> +</ul> + +<p>For example, given the following:</p> +<pre class="highlight plaintext"><code>setup_task = Task( + ... + processes=[download_interpreter, update_zookeeper], + # It is important to note that {{Tasks.concat}} has + # no effect on the ordering of the processes within a task; + # hence the necessity of the {{order}} statement below + # (otherwise, the order in which {{download_interpreter}} + # and {{update_zookeeper}} run will be non-deterministic) + constraints=order(download_interpreter, update_zookeeper), + ... +) + +run_task = SequentialTask( + ... + processes=[download_application, start_application], + ... +) + +combined_task = Tasks.concat(setup_task, run_task) +</code></pre> + +<p>The <code>Tasks.concat</code> command merges the two Tasks into a single Task and +ensures all processes in <code>setup_task</code> run before the processes +in <code>run_task</code>. Conceptually, the task is reduced to:</p> +<pre class="highlight plaintext"><code>task = Task( + ... + processes=[download_interpreter, update_zookeeper, + download_application, start_application], + constraints=order(download_interpreter, update_zookeeper, + download_application, start_application), + ... +) +</code></pre> + +<p>In the case of <code>Tasks.combine</code>, the two schedules run in parallel:</p> +<pre class="highlight plaintext"><code>task = Task( + ... + processes=[download_interpreter, update_zookeeper, + download_application, start_application], + constraints=order(download_interpreter, update_zookeeper) + + order(download_application, start_application), + ... +) +</code></pre> + +<p>In the latter case, each of the two sequences may operate in parallel. +Of course, this may not be the intended behavior (for example, if +the <code>start_application</code> Process implicitly relies +upon <code>download_interpreter</code>). Make sure you understand the difference +between using one or the other.</p> + +<h2 id="defining-job-objects">Defining Job Objects</h2> + +<p>A job is a group of identical tasks that Aurora can run in a Mesos cluster.</p> + +<p>A <code>Job</code> object is defined by the values of several attributes, some +required and some optional. The required attributes are:</p> + +<ul> +<li><p><code>task</code>: Task object to bind to this job. Note that a Job can +only take a single Task.</p></li> +<li><p><code>role</code>: Job’s role account; in other words, the user account to run +the job as on a Mesos cluster machine. A common value is +<code>os.getenv('USER')</code>; using a Python command to get the user who +submits the job request. The other common value is the service +account that runs the job, e.g. <code>www-data</code>.</p></li> +<li><p><code>environment</code>: Job’s environment, typical values +are <code>devel</code>, <code>test</code>, or <code>prod</code>.</p></li> +<li><p><code>cluster</code>: Aurora cluster to schedule the job in, defined in +<code>/etc/aurora/clusters.json</code> or <code>~/.clusters.json</code>. You can specify +jobs where the only difference is the <code>cluster</code>, then at run time +only run the Job whose job key includes your desired cluster’s name.</p></li> +</ul> + +<p>You usually see a <code>name</code> parameter. By default, <code>name</code> inherits its +value from the Job’s associated Task object, but you can override this +default. For these four parameters, a Job definition might look like:</p> +<pre class="highlight plaintext"><code>foo_job = Job( name = 'foo', cluster = 'cluster1', + role = os.getenv('USER'), environment = 'prod', + task = foo_task) +</code></pre> + +<p>In addition to the required attributes, there are several optional +attributes. Details can be found in the <a href="/documentation/0.12.0/configuration-reference/#job-objects">Aurora+Thermos Configuration Reference</a>.</p> + +<h2 id="the-jobs-list">The jobs List</h2> + +<p>At the end of your <code>.aurora</code> file, you need to specify a list of the +file’s defined Jobs. For example, the following exports the jobs <code>job1</code>, +<code>job2</code>, and <code>job3</code>.</p> +<pre class="highlight plaintext"><code>jobs = [job1, job2, job3] +</code></pre> + +<p>This allows the aurora client to invoke commands on those jobs, such as +starting, updating, or killing them.</p> + +<h2 id="templating">Templating</h2> + +<p>The <code>.aurora</code> file format is just Python. However, <code>Job</code>, <code>Task</code>, +<code>Process</code>, and other classes are defined by a templating library called +<em>Pystachio</em>, a powerful tool for configuration specification and reuse.</p> + +<p><a href="/documentation/0.12.0/configuration-reference/">Aurora+Thermos Configuration Reference</a> +has a full reference of all Aurora/Thermos defined Pystachio objects.</p> + +<p>When writing your <code>.aurora</code> file, you may use any Pystachio datatypes, as +well as any objects shown in the <em>Aurora+Thermos Configuration +Reference</em> without <code>import</code> statements - the Aurora config loader +injects them automatically. Other than that the <code>.aurora</code> format +works like any other Python script.</p> + +<h3 id="templating-1-binding-in-pystachio">Templating 1: Binding in Pystachio</h3> + +<p>Pystachio uses the visually distinctive {{}} to indicate template +variables. These are often called “mustache variables” after the +similarly appearing variables in the Mustache templating system and +because the curly braces resemble mustaches.</p> + +<p>If you are familiar with the Mustache system, templates in Pystachio +have significant differences. They have no nesting, joining, or +inheritance semantics. On the other hand, when evaluated, templates +are evaluated iteratively, so this affords some level of indirection.</p> + +<p>Let’s start with the simplest template; text with one +variable, in this case <code>name</code>;</p> +<pre class="highlight plaintext"><code>Hello {{name}} +</code></pre> + +<p>If we evaluate this as is, we’d get back:</p> +<pre class="highlight plaintext"><code>Hello +</code></pre> + +<p>If a template variable doesn’t have a value, when evaluated it’s +replaced with nothing. If we add a binding to give it a value:</p> +<pre class="highlight json"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb"> </span><span style="color: #000080">"name"</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"Tom"</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb"> +</span></code></pre> + +<p>We’d get back:</p> +<pre class="highlight plaintext"><code>Hello Tom +</code></pre> + +<p>Every Pystachio object has an associated <code>.bind</code> method that can bind +values to {{}} variables. Bindings are not immediately evaluated. +Instead, they are evaluated only when the interpolated value of the +object is necessary, e.g. for performing equality or serializing a +message over the wire.</p> + +<p>Objects with and without mustache templated variables behave +differently:</p> +<pre class="highlight plaintext"><code>>>> Float(1.5) +Float(1.5) + +>>> Float('{{x}}.5') +Float({{x}}.5) + +>>> Float('{{x}}.5').bind(x = 1) +Float(1.5) + +>>> Float('{{x}}.5').bind(x = 1) == Float(1.5) +True + +>>> contextual_object = String('{{metavar{{number}}}}').bind( +... metavar1 = "first", metavar2 = "second") + +>>> contextual_object +String({{metavar{{number}}}}) + +>>> contextual_object.bind(number = 1) +String(first) + +>>> contextual_object.bind(number = 2) +String(second) +</code></pre> + +<p>You usually bind simple key to value pairs, but you can also bind three +other objects: lists, dictionaries, and structurals. These will be +described in detail later.</p> + +<h3 id="structurals-in-pystachio-aurora">Structurals in Pystachio / Aurora</h3> + +<p>Most Aurora/Thermos users don’t ever (knowingly) interact with <code>String</code>, +<code>Float</code>, or <code>Integer</code> Pystashio objects directly. Instead they interact +with derived structural (<code>Struct</code>) objects that are collections of +fundamental and structural objects. The structural object components are +called <em>attributes</em>. Aurora’s most used structural objects are <code>Job</code>, +<code>Task</code>, and <code>Process</code>:</p> +<pre class="highlight plaintext"><code>class Process(Struct): + cmdline = Required(String) + name = Required(String) + max_failures = Default(Integer, 1) + daemon = Default(Boolean, False) + ephemeral = Default(Boolean, False) + min_duration = Default(Integer, 5) + final = Default(Boolean, False) +</code></pre> + +<p>Construct default objects by following the object’s type with (). If you +want an attribute to have a value different from its default, include +the attribute name and value inside the parentheses.</p> +<pre class="highlight plaintext"><code>>>> Process() +Process(daemon=False, max_failures=1, ephemeral=False, + min_duration=5, final=False) +</code></pre> + +<p>Attribute values can be template variables, which then receive specific +values when creating the object.</p> +<pre class="highlight plaintext"><code>>>> Process(cmdline = 'echo {{message}}') +Process(daemon=False, max_failures=1, ephemeral=False, min_duration=5, + cmdline=echo {{message}}, final=False) + +>>> Process(cmdline = 'echo {{message}}').bind(message = 'hello world') +Process(daemon=False, max_failures=1, ephemeral=False, min_duration=5, + cmdline=echo hello world, final=False) +</code></pre> + +<p>A powerful binding property is that all of an object’s children inherit its +bindings:</p> +<pre class="highlight plaintext"><code>>>> List(Process)([ +... Process(name = '{{prefix}}_one'), +... Process(name = '{{prefix}}_two') +... ]).bind(prefix = 'hello') +ProcessList( + Process(daemon=False, name=hello_one, max_failures=1, ephemeral=False, min_duration=5, final=False), + Process(daemon=False, name=hello_two, max_failures=1, ephemeral=False, min_duration=5, final=False) + ) +</code></pre> + +<p>Remember that an Aurora Job contains Tasks which contain Processes. A +Job level binding is inherited by its Tasks and all their Processes. +Similarly a Task level binding is available to that Task and its +Processes but is <em>not</em> visible at the Job level (inheritance is a +one-way street.)</p> + +<h4 id="mustaches-within-structurals">Mustaches Within Structurals</h4> + +<p>When you define a <code>Struct</code> schema, one powerful, but confusing, feature +is that all of that structure’s attributes are Mustache variables within +the enclosing scope <em>once they have been populated</em>.</p> + +<p>For example, when <code>Process</code> is defined above, all its attributes such as +{{<code>name</code>}}, {{<code>cmdline</code>}}, {{<code>max_failures</code>}} etc., are all immediately +defined as Mustache variables, implicitly bound into the <code>Process</code>, and +inherit all child objects once they are defined.</p> + +<p>Thus, you can do the following:</p> +<pre class="highlight plaintext"><code>>>> Process(name = "installer", cmdline = "echo {{name}} is running") +Process(daemon=False, name=installer, max_failures=1, ephemeral=False, min_duration=5, + cmdline=echo installer is running, final=False) +</code></pre> + +<p>WARNING: This binding only takes place in one direction. For example, +the following does NOT work and does not set the <code>Process</code> <code>name</code> +attribute’s value.</p> +<pre class="highlight plaintext"><code>>>> Process().bind(name = "installer") +Process(daemon=False, max_failures=1, ephemeral=False, min_duration=5, final=False) +</code></pre> + +<p>The following is also not possible and results in an infinite loop that +attempts to resolve <code>Process.name</code>.</p> +<pre class="highlight plaintext"><code>>>> Process(name = '{{name}}').bind(name = 'installer') +</code></pre> + +<p>Do not confuse Structural attributes with bound Mustache variables. +Attributes are implicitly converted to Mustache variables but not vice +versa.</p> + +<h3 id="templating-2-structurals-are-factories">Templating 2: Structurals Are Factories</h3> + +<h4 id="a-second-way-of-templating">A Second Way of Templating</h4> + +<p>A second templating method is both as powerful as the aforementioned and +often confused with it. This method is due to automatic conversion of +Struct attributes to Mustache variables as described above.</p> + +<p>Suppose you create a Process object:</p> +<pre class="highlight plaintext"><code>>>> p = Process(name = "process_one", cmdline = "echo hello world") + +>>> p +Process(daemon=False, name=process_one, max_failures=1, ephemeral=False, min_duration=5, + cmdline=echo hello world, final=False) +</code></pre> + +<p>This <code>Process</code> object, “<code>p</code>”, can be used wherever a <code>Process</code> object is +needed. It can also be reused by changing the value(s) of its +attribute(s). Here we change its <code>name</code> attribute from <code>process_one</code> to +<code>process_two</code>.</p> +<pre class="highlight plaintext"><code>>>> p(name = "process_two") +Process(daemon=False, name=process_two, max_failures=1, ephemeral=False, min_duration=5, + cmdline=echo hello world, final=False) +</code></pre> + +<p>Template creation is a common use for this technique:</p> +<pre class="highlight plaintext"><code>>>> Daemon = Process(daemon = True) +>>> logrotate = Daemon(name = 'logrotate', cmdline = './logrotate conf/logrotate.conf') +>>> mysql = Daemon(name = 'mysql', cmdline = 'bin/mysqld --safe-mode') +</code></pre> + +<h3 id="advanced-binding">Advanced Binding</h3> + +<p>As described above, <code>.bind()</code> binds simple strings or numbers to +Mustache variables. In addition to Structural types formed by combining +atomic types, Pystachio has two container types; <code>List</code> and <code>Map</code> which +can also be bound via <code>.bind()</code>.</p> + +<h4 id="bind-syntax">Bind Syntax</h4> + +<p>The <code>bind()</code> function can take Python dictionaries or <code>kwargs</code> +interchangeably (when “<code>kwargs</code>” is in a function definition, <code>kwargs</code> +receives a Python dictionary containing all keyword arguments after the +formal parameter list).</p> +<pre class="highlight plaintext"><code>>>> String('{{foo}}').bind(foo = 'bar') == String('{{foo}}').bind({'foo': 'bar'}) +True +</code></pre> + +<p>Bindings done “closer” to the object in question take precedence:</p> +<pre class="highlight plaintext"><code>>>> p = Process(name = '{{context}}_process') +>>> t = Task().bind(context = 'global') +>>> t(processes = [p, p.bind(context = 'local')]) +Task(processes=ProcessList( + Process(daemon=False, name=global_process, max_failures=1, ephemeral=False, final=False, + min_duration=5), + Process(daemon=False, name=local_process, max_failures=1, ephemeral=False, final=False, + min_duration=5) +)) +</code></pre> + +<h4 id="binding-complex-objects">Binding Complex Objects</h4> + +<h5 id="lists">Lists</h5> +<pre class="highlight plaintext"><code>>>> fibonacci = List(Integer)([1, 1, 2, 3, 5, 8, 13]) +>>> String('{{fib[4]}}').bind(fib = fibonacci) +String(5) +</code></pre> + +<h5 id="maps">Maps</h5> +<pre class="highlight plaintext"><code>>>> first_names = Map(String, String)({'Kent': 'Clark', 'Wayne': 'Bruce', 'Prince': 'Diana'}) +>>> String('{{first[Kent]}}').bind(first = first_names) +String(Clark) +</code></pre> + +<h5 id="structurals">Structurals</h5> +<pre class="highlight plaintext"><code>>>> String('{{p.cmdline}}').bind(p = Process(cmdline = "echo hello world")) +String(echo hello world) +</code></pre> + +<h3 id="structural-binding">Structural Binding</h3> + +<p>Use structural templates when binding more than two or three individual +values at the Job or Task level. For fewer than two or three, standard +key to string binding is sufficient.</p> + +<p>Structural binding is a very powerful pattern and is most useful in +Aurora/Thermos for doing Structural configuration. For example, you can +define a job profile. The following profile uses <code>HDFS</code>, the Hadoop +Distributed File System, to designate a file’s location. <code>HDFS</code> does +not come with Aurora, so you’ll need to either install it separately +or change the way the dataset is designated.</p> +<pre class="highlight plaintext"><code>class Profile(Struct): + version = Required(String) + environment = Required(String) + dataset = Default(String, hdfs://home/aurora/data/{{environment}}') + +PRODUCTION = Profile(version = 'live', environment = 'prod') +DEVEL = Profile(version = 'latest', + environment = 'devel', + dataset = 'hdfs://home/aurora/data/test') +TEST = Profile(version = 'latest', environment = 'test') + +JOB_TEMPLATE = Job( + name = 'application', + role = 'myteam', + cluster = 'cluster1', + environment = '{{profile.environment}}', + task = SequentialTask( + name = 'task', + resources = Resources(cpu = 2, ram = 4*GB, disk = 8*GB), + processes = [ + Process(name = 'main', cmdline = 'java -jar application.jar -hdfsPath + {{profile.dataset}}') + ] + ) + ) + +jobs = [ + JOB_TEMPLATE(instances = 100).bind(profile = PRODUCTION), + JOB_TEMPLATE.bind(profile = DEVEL), + JOB_TEMPLATE.bind(profile = TEST), + ] +</code></pre> + +<p>In this case, a custom structural “Profile” is created to self-document +the configuration to some degree. This also allows some schema +“type-checking”, and for default self-substitution, e.g. in +<code>Profile.dataset</code> above.</p> + +<p>So rather than a <code>.bind()</code> with a half-dozen substituted variables, you +can bind a single object that has sensible defaults stored in a single +place.</p> + +<h2 id="configuration-file-writing-tips-and-best-practices">Configuration File Writing Tips And Best Practices</h2> + +<h3 id="use-as-few-aurora-files-as-possible">Use As Few .aurora Files As Possible</h3> + +<p>When creating your <code>.aurora</code> configuration, try to keep all versions of +a particular job within the same <code>.aurora</code> file. For example, if you +have separate jobs for <code>cluster1</code>, <code>cluster1</code> staging, <code>cluster1</code> +testing, and<code>cluster2</code>, keep them as close together as possible.</p> + +<p>Constructs shared across multiple jobs owned by your team (e.g. +team-level defaults or structural templates) can be split into separate +<code>.aurora</code>files and included via the <code>include</code> directive.</p> + +<h3 id="avoid-boilerplate">Avoid Boilerplate</h3> + +<p>If you see repetition or find yourself copy and pasting any parts of +your configuration, it’s likely an opportunity for templating. Take the +example below:</p> + +<p><code>redundant.aurora</code> contains:</p> +<pre class="highlight plaintext"><code>download = Process( + name = 'download', + cmdline = 'wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2', + max_failures = 5, + min_duration = 1) + +unpack = Process( + name = 'unpack', + cmdline = 'rm -rf Python-2.7.3 && tar xzf Python-2.7.3.tar.bz2', + max_failures = 5, + min_duration = 1) + +build = Process( + name = 'build', + cmdline = 'pushd Python-2.7.3 && ./configure && make && popd', + max_failures = 1) + +email = Process( + name = 'email', + cmdline = 'echo Success | mail [email protected]', + max_failures = 5, + min_duration = 1) + +build_python = Task( + name = 'build_python', + processes = [download, unpack, build, email], + constraints = [Constraint(order = ['download', 'unpack', 'build', 'email'])]) +</code></pre> + +<p>As you’ll notice, there’s a lot of repetition in the <code>Process</code> +definitions. For example, almost every process sets a <code>max_failures</code> +limit to 5 and a <code>min_duration</code> to 1. This is an opportunity for factoring +into a common process template.</p> + +<p>Furthermore, the Python version is repeated everywhere. This can be +bound via structural templating as described in the <a href="#AdvancedBinding">Advanced Binding</a> +section.</p> + +<p><code>less_redundant.aurora</code> contains:</p> +<pre class="highlight plaintext"><code>class Python(Struct): + version = Required(String) + base = Default(String, 'Python-{{version}}') + package = Default(String, '{{base}}.tar.bz2') + +ReliableProcess = Process( + max_failures = 5, + min_duration = 1) + +download = ReliableProcess( + name = 'download', + cmdline = 'wget http://www.python.org/ftp/python/{{python.version}}/{{python.package}}') + +unpack = ReliableProcess( + name = 'unpack', + cmdline = 'rm -rf {{python.base}} && tar xzf {{python.package}}') + +build = ReliableProcess( + name = 'build', + cmdline = 'pushd {{python.base}} && ./configure && make && popd', + max_failures = 1) + +email = ReliableProcess( + name = 'email', + cmdline = 'echo Success | mail {{role}}@foocorp.com') + +build_python = SequentialTask( + name = 'build_python', + processes = [download, unpack, build, email]).bind(python = Python(version = "2.7.3")) +</code></pre> + +<h3 id="thermos-uses-bash-but-thermos-is-not-bash">Thermos Uses bash, But Thermos Is Not bash</h3> + +<h4 id="bad">Bad</h4> + +<p>Many tiny Processes makes for harder to manage configurations.</p> +<pre class="highlight plaintext"><code>copy = Process( + name = 'copy', + cmdline = 'rcp user@my_machine:my_application .' + ) + + unpack = Process( + name = 'unpack', + cmdline = 'unzip app.zip' + ) + + remove = Process( + name = 'remove', + cmdline = 'rm -f app.zip' + ) + + run = Process( + name = 'app', + cmdline = 'java -jar app.jar' + ) + + run_task = Task( + processes = [copy, unpack, remove, run], + constraints = order(copy, unpack, remove, run) + ) +</code></pre> + +<h4 id="good">Good</h4> + +<p>Each <code>cmdline</code> runs in a bash subshell, so you have the full power of +bash. Chaining commands with <code>&&</code> or <code>||</code> is almost always the right +thing to do.</p> + +<p>Also for Tasks that are simply a list of processes that run one after +another, consider using the <code>SequentialTask</code> helper which applies a +linear ordering constraint for you.</p> +<pre class="highlight plaintext"><code>stage = Process( + name = 'stage', + cmdline = 'rcp user@my_machine:my_application . && unzip app.zip && rm -f app.zip') + +run = Process(name = 'app', cmdline = 'java -jar app.jar') + +run_task = SequentialTask(processes = [stage, run]) +</code></pre> + +<h3 id="rarely-use-functions-in-your-configurations">Rarely Use Functions In Your Configurations</h3> + +<p>90% of the time you define a function in a <code>.aurora</code> file, you’re +probably Doing It Wrong™.</p> + +<h4 id="bad">Bad</h4> +<pre class="highlight plaintext"><code>def get_my_task(name, user, cpu, ram, disk): + return Task( + name = name, + user = user, + processes = [STAGE_PROCESS, RUN_PROCESS], + constraints = order(STAGE_PROCESS, RUN_PROCESS), + resources = Resources(cpu = cpu, ram = ram, disk = disk) + ) + + task_one = get_my_task('task_one', 'feynman', 1.0, 32*MB, 1*GB) + task_two = get_my_task('task_two', 'feynman', 2.0, 64*MB, 1*GB) +</code></pre> + +<h4 id="good">Good</h4> + +<p>This one is more idiomatic. Forced keyword arguments prevents accidents, +e.g. constructing a task with “32*MB” when you mean 32MB of ram and not +disk. Less proliferation of task-construction techniques means +easier-to-read, quicker-to-understand, and a more composable +configuration.</p> +<pre class="highlight plaintext"><code>TASK_TEMPLATE = SequentialTask( + user = 'wickman', + processes = [STAGE_PROCESS, RUN_PROCESS], +) + +task_one = TASK_TEMPLATE( + name = 'task_one', + resources = Resources(cpu = 1.0, ram = 32*MB, disk = 1*GB) ) + +task_two = TASK_TEMPLATE( + name = 'task_two', + resources = Resources(cpu = 2.0, ram = 64*MB, disk = 1*GB) +) +</code></pre> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
Added: aurora/site/publish/documentation/0.12.0/cron-jobs/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.12.0/cron-jobs/index.html?rev=1733548&view=auto ============================================================================== --- aurora/site/publish/documentation/0.12.0/cron-jobs/index.html (added) +++ aurora/site/publish/documentation/0.12.0/cron-jobs/index.html Fri Mar 4 02:43:01 2016 @@ -0,0 +1,259 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/cron-jobs/'" + value="0.12.0"> + <option value="0.12.0" + selected="selected"> + 0.12.0 + (latest) + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="cron-jobs">Cron Jobs</h1> + +<p>Aurora supports execution of scheduled jobs on a Mesos cluster using cron-style syntax.</p> + +<ul> +<li><a href="#overview">Overview</a></li> +<li><a href="#collision-policies">Collision Policies</a> + +<ul> +<li><a href="#kill_existing">KILL_EXISTING</a></li> +<li><a href="#cancel_new">CANCEL_NEW</a></li> +</ul></li> +<li><a href="#failure-recovery">Failure recovery</a></li> +<li><a href="#interacting-with-cron-jobs-via-the-aurora-cli">Interacting with cron jobs via the Aurora CLI</a> + +<ul> +<li><a href="#cron-schedule">cron schedule</a></li> +<li><a href="#cron-deschedule">cron deschedule</a></li> +<li><a href="#cron-start">cron start</a></li> +<li><a href="#job-killall-job-restart-job-kill">job killall, job restart, job kill</a></li> +</ul></li> +<li><a href="#technical-note-about-syntax">Technical Note About Syntax</a></li> +<li><a href="#caveats">Caveats</a> + +<ul> +<li><a href="#failovers">Failovers</a></li> +<li><a href="#collision-policy-is-best-effort">Collision policy is best-effort</a></li> +<li><a href="#timezone-configuration">Timezone Configuration</a></li> +</ul></li> +</ul> + +<h2 id="overview">Overview</h2> + +<p>A job is identified as a cron job by the presence of a +<code>cron_schedule</code> attribute containing a cron-style schedule in the +<a href="/documentation/0.12.0/configuration-reference/#job-objects"><code>Job</code></a> object. Examples of cron schedules +include “every 5 minutes” (<code>*/5 * * * *</code>), “Fridays at 17:00” (<code>* 17 * * FRI</code>), and +“the 1st and 15th day of the month at 03:00” (<code>0 3 1,15 *</code>).</p> + +<p>Example (available in the <a href="/documentation/0.12.0/vagrant/">Vagrant environment</a>):</p> +<pre class="highlight plaintext"><code>$ cat /vagrant/examples/job/cron_hello_world.aurora +# cron_hello_world.aurora +# A cron job that runs every 5 minutes. +jobs = [ + Job( + cluster = 'devcluster', + role = 'www-data', + environment = 'test', + name = 'cron_hello_world', + cron_schedule = '*/5 * * * *', + task = SimpleTask( + 'cron_hello_world', + 'echo "Hello world from cron, the time is now $(date --rfc-822)"'), + ), +] +</code></pre> + +<h2 id="collision-policies">Collision Policies</h2> + +<p>The <code>cron_collision_policy</code> field specifies the scheduler’s behavior when a new cron job is +triggered while an older run hasn’t finished. The scheduler has two policies available, +<a href="#kill_existing">KILL_EXISTING</a> and <a href="#cancel_new">CANCEL_NEW</a>.</p> + +<h3 id="kill_existing">KILL_EXISTING</h3> + +<p>The default policy - on a collision the old instances are killed and a instances with the current +configuration are started.</p> + +<h3 id="cancel_new">CANCEL_NEW</h3> + +<p>On a collision the new run is cancelled.</p> + +<p>Note that the use of this flag is likely a code smell - interrupted cron jobs should be able +to recover their progress on a subsequent invocation, otherwise they risk having their work queue +grow faster than they can process it.</p> + +<h2 id="failure-recovery">Failure recovery</h2> + +<p>Unlike with services, which aurora will always re-execute regardless of exit status, instances of +cron jobs retry according to the <code>max_task_failures</code> attribute of the +<a href="/documentation/0.12.0/configuration-reference/#task-objects">Task</a> object. To get “run-until-success” semantics, +set <code>max_task_failures</code> to <code>-1</code>.</p> + +<h2 id="interacting-with-cron-jobs-via-the-aurora-cli">Interacting with cron jobs via the Aurora CLI</h2> + +<p>Most interaction with cron jobs takes place using the <code>cron</code> subcommand. See <code>aurora cron -h</code> +for up-to-date usage instructions.</p> + +<h3 id="cron-schedule">cron schedule</h3> + +<p>Schedules a new cron job on the Aurora cluster for later runs or replaces the existing cron template +with a new one. Only future runs will be affected, any existing active tasks are left intact.</p> +<pre class="highlight plaintext"><code>$ aurora cron schedule devcluster/www-data/test/cron_hello_world /vagrant/examples/jobs/cron_hello_world.aurora +</code></pre> + +<h3 id="cron-deschedule">cron deschedule</h3> + +<p>Deschedules a cron job, preventing future runs but allowing current runs to complete.</p> +<pre class="highlight plaintext"><code>$ aurora cron deschedule devcluster/www-data/test/cron_hello_world +</code></pre> + +<h3 id="cron-start">cron start</h3> + +<p>Start a cron job immediately, outside of its normal cron schedule.</p> +<pre class="highlight plaintext"><code>$ aurora cron start devcluster/www-data/test/cron_hello_world +</code></pre> + +<h3 id="job-killall-job-restart-job-kill">job killall, job restart, job kill</h3> + +<p>Cron jobs create instances running on the cluster that you can interact with like normal Aurora +tasks with <code>job kill</code> and <code>job restart</code>.</p> + +<h2 id="technical-note-about-syntax">Technical Note About Syntax</h2> + +<p><code>cron_schedule</code> uses a restricted subset of BSD crontab syntax. While the +execution engine currently uses Quartz, the schedule parsing is custom, a subset of FreeBSD +<a href="http://www.freebsd.org/cgi/man.cgi?crontab(5)">crontab(5)</a> syntax. See +<a href="https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/cron/CrontabEntry.java#L106-L124";>the source</a> +for details.</p> + +<h2 id="caveats">Caveats</h2> + +<h3 id="failovers">Failovers</h3> + +<p>No failover recovery. Aurora does not record the latest minute it fired +triggers for across failovers. Therefore it’s possible to miss triggers +on failover. Note that this behavior may change in the future.</p> + +<p>It’s necessary to sync time between schedulers with something like <code>ntpd</code>. +Clock skew could cause double or missed triggers in the case of a failover.</p> + +<h3 id="collision-policy-is-best-effort">Collision policy is best-effort</h3> + +<p>Aurora aims to always have <em>at least one copy</em> of a given instance running at a time - it’s +an AP system, meaning it chooses Availability and Partition Tolerance at the expense of +Consistency.</p> + +<p>If your collision policy was <code>CANCEL_NEW</code> and a task has terminated but +Aurora has not noticed this Aurora will go ahead and create your new +task.</p> + +<p>If your collision policy was <code>KILL_EXISTING</code> and a task was marked <code>LOST</code> +but not yet GCed Aurora will go ahead and create your new task without +attempting to kill the old one (outside the GC interval).</p> + +<h3 id="timezone-configuration">Timezone Configuration</h3> + +<p>Cron timezone is configured indepdendently of JVM timezone with the <code>-cron_timezone</code> flag and +defaults to UTC.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.12.0/deploying-aurora-scheduler/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.12.0/deploying-aurora-scheduler/index.html?rev=1733548&view=auto ============================================================================== --- aurora/site/publish/documentation/0.12.0/deploying-aurora-scheduler/index.html (added) +++ aurora/site/publish/documentation/0.12.0/deploying-aurora-scheduler/index.html Fri Mar 4 02:43:01 2016 @@ -0,0 +1,548 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/deploying-aurora-scheduler/'" + value="0.12.0"> + <option value="0.12.0" + selected="selected"> + 0.12.0 + (latest) + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="deploying-the-aurora-scheduler">Deploying the Aurora Scheduler</h1> + +<p>When setting up your cluster, you will install the scheduler on a small number (usually 3 or 5) of +machines. This guide helps you get the scheduler set up and troubleshoot some common hurdles.</p> + +<ul> +<li><a href="#installing-aurora">Installing Aurora</a> + +<ul> +<li><a href="#creating-the-distribution-zip-file-optional">Creating the Distribution .zip File (Optional)</a></li> +<li><a href="#installing-aurora-1">Installing Aurora</a></li> +</ul></li> +<li><a href="#configuring-aurora">Configuring Aurora</a> + +<ul> +<li><a href="#a-note-on-configuration">A Note on Configuration</a></li> +<li><a href="#replicated-log-configuration">Replicated Log Configuration</a></li> +<li><a href="#initializing-the-replicated-log">Initializing the Replicated Log</a></li> +<li><a href="#storage-performance-considerations">Storage Performance Considerations</a></li> +<li><a href="#network-considerations">Network considerations</a></li> +<li><a href="#considerations-for-running-jobs-in-docker">Considerations for running jobs in docker</a></li> +<li><a href="#security-considerations">Security Considerations</a></li> +<li><a href="#configuring-resource-oversubscription">Configuring Resource Oversubscription</a></li> +<li><a href="#process-logs">Process Logs</a></li> +</ul></li> +<li><a href="#running-aurora">Running Aurora</a> + +<ul> +<li><a href="#maintaining-an-aurora-installation">Maintaining an Aurora Installation</a></li> +<li><a href="#monitoring">Monitoring</a></li> +<li><a href="#running-stateful-services">Running stateful services</a></li> +<li><a href="#dedicated-attribute">Dedicated attribute</a> + +<ul> +<li><a href="#syntax">Syntax</a></li> +<li><a href="#example">Example</a></li> +</ul></li> +</ul></li> +<li><a href="#best-practices">Best practices</a> + +<ul> +<li><a href="#diversity">Diversity</a></li> +</ul></li> +<li><a href="#common-problems">Common problems</a> + +<ul> +<li><a href="#replicated-log-not-initialized">Replicated log not initialized</a></li> +<li><a href="#symptoms">Symptoms</a></li> +<li><a href="#solution">Solution</a></li> +<li><a href="#scheduler-not-registered">Scheduler not registered</a></li> +<li><a href="#symptoms-1">Symptoms</a></li> +<li><a href="#solution-1">Solution</a></li> +</ul></li> +<li><a href="#changing-scheduler-quorum-size">Changing Scheduler Quorum Size</a> + +<ul> +<li><a href="#preparation">Preparation</a></li> +<li><a href="#adding-new-schedulers">Adding New Schedulers</a></li> +</ul></li> +</ul> + +<h2 id="installing-aurora">Installing Aurora</h2> + +<p>The Aurora scheduler is a standalone Java server. As part of the build process it creates a bundle +of all its dependencies, with the notable exceptions of the JVM and libmesos. Each target server +should have a JVM (Java 8 or higher) and libmesos (0.25.0) installed.</p> + +<h3 id="creating-the-distribution-zip-file-optional">Creating the Distribution .zip File (Optional)</h3> + +<p>To create a distribution for installation you will need build tools installed. On Ubuntu this can be +done with <code>sudo apt-get install build-essential default-jdk</code>.</p> +<pre class="highlight plaintext"><code>git clone http://git-wip-us.apache.org/repos/asf/aurora.git +cd aurora +./gradlew distZip +</code></pre> + +<p>Copy the generated <code>dist/distributions/aurora-scheduler-*.zip</code> to each node that will run a scheduler.</p> + +<h3 id="installing-aurora">Installing Aurora</h3> + +<p>Extract the aurora-scheduler zip file. The example configurations assume it is extracted to +<code>/usr/local/aurora-scheduler</code>.</p> +<pre class="highlight plaintext"><code>sudo unzip dist/distributions/aurora-scheduler-*.zip -d /usr/local +sudo ln -nfs "$(ls -dt /usr/local/aurora-scheduler-* | head -1)" /usr/local/aurora-scheduler +</code></pre> + +<h2 id="configuring-aurora">Configuring Aurora</h2> + +<h3 id="a-note-on-configuration">A Note on Configuration</h3> + +<p>Like Mesos, Aurora uses command-line flags for runtime configuration. As such the Aurora +“configuration file” is typically a <code>scheduler.sh</code> shell script of the form.</p> +<pre class="highlight shell"><code><span style="color: #999988;font-style: italic">#!/bin/bash</span> +<span style="color: #008080">AURORA_HOME</span><span style="color: #000000;font-weight: bold">=</span>/usr/local/aurora-scheduler + +<span style="color: #999988;font-style: italic"># Flags controlling the JVM.</span> +<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=(</span> + -Xmx2g + -Xms2g + <span style="color: #999988;font-style: italic"># GC tuning, etc.</span> +<span style="color: #000000;font-weight: bold">)</span> + +<span style="color: #999988;font-style: italic"># Flags controlling the scheduler.</span> +<span style="color: #008080">AURORA_FLAGS</span><span style="color: #000000;font-weight: bold">=(</span> + -http_port<span style="color: #000000;font-weight: bold">=</span>8081 + <span style="color: #999988;font-style: italic"># Log configuration, etc.</span> +<span style="color: #000000;font-weight: bold">)</span> + +<span style="color: #999988;font-style: italic"># Environment variables controlling libmesos</span> +<span style="color: #0086B3">export </span><span style="color: #008080">JAVA_HOME</span><span style="color: #000000;font-weight: bold">=</span>... +<span style="color: #0086B3">export </span><span style="color: #008080">GLOG_v</span><span style="color: #000000;font-weight: bold">=</span>1 +<span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_PORT</span><span style="color: #000000;font-weight: bold">=</span>8083 + +<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">JAVA_OPTS</span><span style="background-color: #f8f8f8">[*]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> <span style="color: #0086B3">exec</span> <span style="color: #d14">"</span><span style="color: #008080">$AURORA_HOME</span><span style="color: #d14">/bin/aurora-scheduler"</span> <span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">AURORA_FLAGS</span><span style="background-color: #f8f8f8">[@]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> +</code></pre> + +<p>That way Aurora’s current flags are visible in <code>ps</code> and in the <code>/vars</code> admin endpoint.</p> + +<p>Examples are available under <code>examples/scheduler/</code>. For a list of available Aurora flags and their +documentation run</p> +<pre class="highlight plaintext"><code>/usr/local/aurora-scheduler/bin/aurora-scheduler -help +</code></pre> + +<h3 id="replicated-log-configuration">Replicated Log Configuration</h3> + +<p>All Aurora state is persisted to a replicated log. This includes all jobs Aurora is running +including where in the cluster they are being run and the configuration for running them, as +well as other information such as metadata needed to reconnect to the Mesos master, resource +quotas, and any other locks in place.</p> + +<p>Aurora schedulers use ZooKeeper to discover log replicas and elect a leader. Only one scheduler is +leader at a given time - the other schedulers follow log writes and prepare to take over as leader +but do not communicate with the Mesos master. Either 3 or 5 schedulers are recommended in a +production deployment depending on failure tolerance and they must have persistent storage.</p> + +<p>In a cluster with <code>N</code> schedulers, the flag <code>-native_log_quorum_size</code> should be set to +<code>floor(N/2) + 1</code>. So in a cluster with 1 scheduler it should be set to <code>1</code>, in a cluster with 3 it +should be set to <code>2</code>, and in a cluster of 5 it should be set to <code>3</code>.</p> + +<table><thead> +<tr> +<th>Number of schedulers (N)</th> +<th><code>-native_log_quorum_size</code> setting (<code>floor(N/2) + 1</code>)</th> +</tr> +</thead><tbody> +<tr> +<td>1</td> +<td>1</td> +</tr> +<tr> +<td>3</td> +<td>2</td> +</tr> +<tr> +<td>5</td> +<td>3</td> +</tr> +<tr> +<td>7</td> +<td>4</td> +</tr> +</tbody></table> + +<p><em>Incorrectly setting this flag will cause data corruption to occur!</em></p> + +<p>See <a href="/documentation/0.12.0/storage-config/#scheduler-storage-configuration-flags">this document</a> for more replicated +log and storage configuration options.</p> + +<h2 id="initializing-the-replicated-log">Initializing the Replicated Log</h2> + +<p>Before you start Aurora you will also need to initialize the log on a majority of the schedulers.</p> +<pre class="highlight plaintext"><code>mesos-log initialize --path="/path/to/native/log" +</code></pre> + +<p>The <code>--path</code> flag should match the <code>--native_log_file_path</code> flag to the scheduler. +Failing to do this will result the following message when you try to start the scheduler.</p> +<pre class="highlight plaintext"><code>Replica in EMPTY status received a broadcasted recover request +</code></pre> + +<h3 id="storage-performance-considerations">Storage Performance Considerations</h3> + +<p>See <a href="/documentation/0.12.0/scheduler-storage/">this document</a> for scheduler storage performance considerations.</p> + +<h3 id="network-considerations">Network considerations</h3> + +<p>The Aurora scheduler listens on 2 ports - an HTTP port used for client RPCs and a web UI, +and a libprocess (HTTP+Protobuf) port used to communicate with the Mesos master and for the log +replication protocol. These can be left unconfigured (the scheduler publishes all selected ports +to ZooKeeper) or explicitly set in the startup script as follows:</p> +<pre class="highlight plaintext"><code># ... +AURORA_FLAGS=( + # ... + -http_port=8081 + # ... +) +# ... +export LIBPROCESS_PORT=8083 +# ... +</code></pre> + +<h3 id="considerations-for-running-jobs-in-docker-containers">Considerations for running jobs in docker containers</h3> + +<p>In order for Aurora to launch jobs using docker containers, a few extra configuration options +must be set. The <a href="http://mesos.apache.org/documentation/latest/docker-containerizer/";>docker containerizer</a> +must be enabled on the mesos slaves by launching them with the <code>--containerizers=docker,mesos</code> option.</p> + +<p>By default, Aurora will configure Mesos to copy the file specified in <code>-thermos_executor_path</code> +into the container’s sandbox. If using a wrapper script to launch the thermos executor, +specify the path to the wrapper in that argument. In addition, the path to the executor pex itself +must be included in the <code>-thermos_executor_resources</code> option. Doing so will ensure that both the +wrapper script and executor are correctly copied into the sandbox. Finally, ensure the wrapper +script does not access resources outside of the sandbox, as when the script is run from within a +docker container those resources will not exist.</p> + +<p>In order to correctly execute processes inside a job, the docker container must have python 2.7 +installed.</p> + +<p>A scheduler flag, <code>-global_container_mounts</code> allows mounting paths from the host (i.e., the slave) +into all containers on that host. The format is a comma separated list of host<em>path:container</em>path[:mode] +tuples. For example <code>-global_container_mounts=/opt/secret_keys_dir:/mnt/secret_keys_dir:ro</code> mounts +<code>/opt/secret_keys_dir</code> from the slaves into all launched containers. Valid modes are <code>ro</code> and <code>rw</code>.</p> + +<p>If you would like to supply your own parameters to <code>docker run</code> when launching jobs in docker +containers, you may use the following flags:</p> +<pre class="highlight plaintext"><code>-allow_docker_parameters +-default_docker_parameters +</code></pre> + +<p><code>-allow_docker_parameters</code> controls whether or not users may pass their own configuration parameters +through the job configuration files. If set to <code>false</code> (the default), the scheduler will reject +jobs with custom parameters. <em>NOTE</em>: this setting should be used with caution as it allows any job +owner to specify any parameters they wish, including those that may introduce security concerns +(<code>privileged=true</code>, for example).</p> + +<p><code>-default_docker_parameters</code> allows a cluster operator to specify a universal set of parameters that +should be used for every container that does not have parameters explicitly configured at the job +level. The argument accepts a multimap format:</p> +<pre class="highlight plaintext"><code>-default_docker_parameters="read-only=true,tmpfs=/tmp,tmpfs=/run" +</code></pre> + +<h3 id="process-logs">Process Logs</h3> + +<h4 id="log-destination">Log destination</h4> + +<p>By default, Thermos will write process stdout/stderr to log files in the sandbox. Process object configuration +allows specifying alternate log file destinations like streamed stdout/stderr or suppression of all log output. +Default behavior can be configured for the entire cluster with the following flag (through the <code>-thermos_executor_flags</code> +argument to the Aurora scheduler):</p> +<pre class="highlight plaintext"><code>--runner-logger-destination=both +</code></pre> + +<p><code>both</code> configuration will send logs to files and stream to parent stdout/stderr outputs.</p> + +<p>See <a href="/documentation/0.12.0/configuration-reference/#logger">this document</a> for all destination options.</p> + +<h4 id="log-rotation">Log rotation</h4> + +<p>By default, Thermos will not rotate the stdout/stderr logs from child processes and they will grow +without bound. An individual user may change this behavior via configuration on the Process object, +but it may also be desirable to change the default configuration for the entire cluster. +In order to enable rotation by default, the following flags can be applied to Thermos (through the +-thermos<em>executor</em>flags argument to the Aurora scheduler):</p> +<pre class="highlight plaintext"><code>--runner-logger-mode=rotate +--runner-rotate-log-size-mb=100 +--runner-rotate-log-backups=10 +</code></pre> + +<p>In the above example, each instance of the Thermos runner will rotate stderr/stdout logs once they +reach 100 MiB in size and keep a maximum of 10 backups. If a user has provided a custom setting for +their process, it will override these default settings.</p> + +<h2 id="running-aurora">Running Aurora</h2> + +<p>Configure a supervisor like <a href="http://mmonit.com/monit/";>Monit</a> or +<a href="http://supervisord.org/";>supervisord</a> to run the created <code>scheduler.sh</code> file and restart it +whenever it fails. Aurora expects to be restarted by an external process when it fails. Aurora +supports an active health checking protocol on its admin HTTP interface - if a <code>GET /health</code> times +out or returns anything other than <code>200 OK</code> the scheduler process is unhealthy and should be +restarted.</p> + +<p>For example, monit can be configured with</p> +<pre class="highlight plaintext"><code>if failed port 8081 send "GET /health HTTP/1.0\r\n" expect "OK\n" with timeout 2 seconds for 10 cycles then restart +</code></pre> + +<p>assuming you set <code>-http_port=8081</code>.</p> + +<h2 id="security-considerations">Security Considerations</h2> + +<p>See <a href="/documentation/0.12.0/security/">security.md</a>.</p> + +<h2 id="configuring-resource-oversubscription">Configuring Resource Oversubscription</h2> + +<p><strong>WARNING</strong>: This feature is currently in alpha status. Do not use it in production clusters! +See <a href="/documentation/0.12.0/configuration-reference/#revocable-jobs">this document</a> for more feature details.</p> + +<p>Set these scheduler flag to allow receiving revocable Mesos offers:</p> +<pre class="highlight plaintext"><code>-receive_revocable_resources=true +</code></pre> + +<p>Specify a tier configuration file path:</p> +<pre class="highlight plaintext"><code>-tier_config=path/to/tiers/config.json +</code></pre> + +<p>Example <a href="https://github.com/apache/aurora/blob/#{git_tag}/src/test/resources/org/apache/aurora/scheduler/tiers-example.json";>tier configuration file</a>).</p> + +<h3 id="maintaining-an-aurora-installation">Maintaining an Aurora Installation</h3> + +<h3 id="monitoring">Monitoring</h3> + +<p>Please see our dedicated <a href="/documentation/0.12.0/monitoring/">monitoring guide</a> for in-depth discussion on monitoring.</p> + +<h3 id="running-stateful-services">Running stateful services</h3> + +<p>Aurora is best suited to run stateless applications, but it also accommodates for stateful services +like databases, or services that otherwise need to always run on the same machines.</p> + +<h4 id="dedicated-attribute">Dedicated attribute</h4> + +<p>The Mesos slave has the <code>--attributes</code> command line argument which can be used to mark a slave with +static attributes (not to be confused with <code>--resources</code>, which are dynamic and accounted).</p> + +<p>Aurora makes these attributes available for matching with scheduling +<a href="/documentation/0.12.0/configuration-reference/#specifying-scheduling-constraints">constraints</a>. Most of these +constraints are arbitrary and available for custom use. There is one exception, though: the +<code>dedicated</code> attribute. Aurora treats this specially, and only allows matching jobs to run on these +machines, and will only schedule matching jobs on these machines.</p> + +<p>See the <a href="/documentation/0.12.0/resources/#resource-quota">section</a> about resource quotas to learn how quotas apply to +dedicated jobs.</p> + +<h5 id="syntax">Syntax</h5> + +<p>The dedicated attribute has semantic meaning. The format is <code>$role(/.*)?</code>. When a job is created, +the scheduler requires that the <code>$role</code> component matches the <code>role</code> field in the job +configuration, and will reject the job creation otherwise. The remainder of the attribute is +free-form. We’ve developed the idiom of formatting this attribute as <code>$role/$job</code>, but do not +enforce this.</p> + +<h5 id="example">Example</h5> + +<p>Consider the following slave command line:</p> +<pre class="highlight plaintext"><code>mesos-slave --attributes="dedicated:db_team/redis" ... +</code></pre> + +<p>And this job configuration:</p> +<pre class="highlight plaintext"><code>Service( + name = 'redis', + role = 'db_team', + constraints = { + 'dedicated': 'db_team/redis' + } + ... +) +</code></pre> + +<p>The job configuration is indicating that it should only be scheduled on slaves with the attribute +<code>dedicated:db_team/redis</code>. Additionally, Aurora will prevent any tasks that do <em>not</em> have that +constraint from running on those slaves.</p> + +<h2 id="best-practices">Best practices</h2> + +<h3 id="diversity">Diversity</h3> + +<p>Data centers are often organized with hierarchical failure domains. Common failure domains +include hosts, racks, rows, and PDUs. If you have this information available, it is wise to tag +the mesos-slave with them as +<a href="https://mesos.apache.org/documentation/attributes-resources/";>attributes</a>.</p> + +<p>When it comes time to schedule jobs, Aurora will automatically spread them across the failure +domains as specified in the +<a href="/documentation/0.12.0/configuration-reference/#specifying-scheduling-constraints">job configuration</a>.</p> + +<p>Note: in virtualized environments like EC2, the only attribute that usually makes sense for this +purpose is <code>host</code>.</p> + +<h2 id="common-problems">Common problems</h2> + +<p>So you’ve started your first cluster and are running into some issues? We’ve collected some common +stumbling blocks and solutions here to help get you moving.</p> + +<h3 id="replicated-log-not-initialized">Replicated log not initialized</h3> + +<h4 id="symptoms">Symptoms</h4> + +<ul> +<li>Scheduler RPCs and web interface claim <code>Storage is not READY</code></li> +<li>Scheduler log repeatedly prints messages like</li> +</ul> +<pre class="highlight plaintext"><code> I1016 16:12:27.234133 26081 replica.cpp:638] Replica in EMPTY status + received a broadcasted recover request + I1016 16:12:27.234256 26084 recover.cpp:188] Received a recover response + from a replica in EMPTY status +</code></pre> + +<h4 id="solution">Solution</h4> + +<p>When you create a new cluster, you need to inform a quorum of schedulers that they are safe to +consider their database to be empty by <a href="#initializing-the-replicated-log">initializing</a> the +replicated log. This is done to prevent the scheduler from modifying the cluster state in the event +of multiple simultaneous disk failures or, more likely, misconfiguration of the replicated log path.</p> + +<h3 id="scheduler-not-registered">Scheduler not registered</h3> + +<h4 id="symptoms">Symptoms</h4> + +<p>Scheduler log contains</p> +<pre class="highlight plaintext"><code>Framework has not been registered within the tolerated delay. +</code></pre> + +<h4 id="solution">Solution</h4> + +<p>Double-check that the scheduler is configured correctly to reach the master. If you are registering +the master in ZooKeeper, make sure command line argument to the master:</p> +<pre class="highlight plaintext"><code>--zk=zk://$ZK_HOST:2181/mesos/master +</code></pre> + +<p>is the same as the one on the scheduler:</p> +<pre class="highlight plaintext"><code>-mesos_master_address=zk://$ZK_HOST:2181/mesos/master +</code></pre> + +<h2 id="changing-scheduler-quorum-size">Changing Scheduler Quorum Size</h2> + +<p>Special care needs to be taken when changing the size of the Aurora scheduler quorum. +Since Aurora uses a Mesos replicated log, similar steps need to be followed as when +<a href="http://mesos.apache.org/documentation/latest/operational-guide";>changing the mesos quorum size</a>.</p> + +<h3 id="preparation">Preparation</h3> + +<p>Increase <a href="/documentation/0.12.0/storage-config/#-native_log_quorum_size">-native<em>log</em>quorum_size</a> on each +existing scheduler and restart them. When updating from 3 to 5 schedulers, the quorum size +would grow from 2 to 3.</p> + +<h3 id="adding-new-schedulers">Adding New Schedulers</h3> + +<p>Start the new schedulers with <code>-native_log_quorum_size</code> set to the new value. Failing to +first increase the quorum size on running schedulers can in some cases result in corruption +or truncating of the replicated log used by Aurora. In that case, see the documentation on +<a href="/documentation/0.12.0/storage-config/#recovering-from-a-scheduler-backup">recovering from backup</a>.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
