Added: aurora/site/publish/documentation/0.17.0/operations/backup-restore/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/operations/backup-restore/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/operations/backup-restore/index.html (added) +++ aurora/site/publish/documentation/0.17.0/operations/backup-restore/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,230 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/operations/backup-restore/'" + value="0.17.0"> + <option value="0.17.0" + selected="selected"> + 0.17.0 + (latest) + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="recovering-from-a-scheduler-backup">Recovering from a Scheduler Backup</h1> + +<p><strong>Be sure to read the entire page before attempting to restore from a backup, as it may have +unintended consequences.</strong></p> + +<h1 id="summary">Summary</h1> + +<p>The restoration procedure replaces the existing (possibly corrupted) Mesos replicated log with an +earlier, backed up, version and requires all schedulers to be taken down temporarily while +restoring. Once completed, the scheduler state resets to what it was when the backup was created. +This means any jobs/tasks created or updated after the backup are unknown to the scheduler and will +be killed shortly after the cluster restarts. All other tasks continue operating as normal.</p> + +<p>Usually, it is a bad idea to restore a backup that is not extremely recent (i.e. older than a few +hours). This is because the scheduler will expect the cluster to look exactly as the backup does, +so any tasks that have been rescheduled since the backup was taken will be killed.</p> + +<p>Instructions below have been verified in <a href="../../getting-started/vagrant/">Vagrant environment</a> and with minor +syntax/path changes should be applicable to any Aurora cluster.</p> + +<h1 id="preparation">Preparation</h1> + +<p>Follow these steps to prepare the cluster for restoring from a backup:</p> + +<ul> +<li><p>Stop all scheduler instances</p></li> +<li><p>Consider blocking external traffic on a port defined in <code>-http_port</code> for all schedulers to +prevent users from interacting with the scheduler during the restoration process. This will help +troubleshooting by reducing the scheduler log noise and prevent users from making changes that will +be erased after the backup snapshot is restored.</p></li> +<li><p>Configure <code>aurora_admin</code> access to run all commands listed in +<a href="#restore-from-backup">Restore from backup</a> section locally on the leading scheduler:</p> + +<ul> +<li>Make sure the <a href="../../reference/client-cluster-configuration/">clusters.json</a> file configured to +access scheduler directly. Set <code>scheduler_uri</code> setting and remove <code>zk</code>. Since leader can get +re-elected during the restore steps, consider doing it on all scheduler replicas.</li> +<li><p>Depending on your particular security approach you will need to either turn off scheduler +authorization by removing scheduler <code>-http_authentication_mechanism</code> flag or make sure the +direct scheduler access is properly authorized. E.g.: in case of Kerberos you will need to make +a <code>/etc/hosts</code> file change to match your local IP to the scheduler URL configured in keytabs:</p> + +<p><local_ip> <scheduler_domain_in_keytabs></p></li> +</ul></li> +<li><p>Next steps are required to put scheduler into a partially disabled state where it would still be +able to accept storage recovery requests but unable to schedule or change task states. This may be +accomplished by updating the following scheduler configuration options:</p> + +<ul> +<li>Set <code>-mesos_master_address</code> to a non-existent zk address. This will prevent scheduler from +registering with Mesos. E.g.: <code>-mesos_master_address=zk://localhost:1111/mesos/master</code></li> +<li><code>-max_registration_delay</code> - set to sufficiently long interval to prevent registration timeout +and as a result scheduler suicide. E.g: <code>-max_registration_delay=360mins</code></li> +<li>Make sure <code>-reconciliation_initial_delay</code> option is set high enough (e.g.: <code>365days</code>) to +prevent accidental task GC. This is important as scheduler will attempt to reconcile the cluster +state and will kill all tasks when restarted with an empty Mesos replicated log.</li> +</ul></li> +<li><p>Restart all schedulers</p></li> +</ul> + +<h1 id="cleanup-and-re-initialize-mesos-replicated-log">Cleanup and re-initialize Mesos replicated log</h1> + +<p>Get rid of the corrupted files and re-initialize Mesos replicated log:</p> + +<ul> +<li>Stop schedulers</li> +<li>Delete all files under <code>-native_log_file_path</code> on all schedulers</li> +<li>Initialize Mesos replica’s log file: <code>sudo mesos-log initialize --path=<-native_log_file_path></code></li> +<li>Start schedulers</li> +</ul> + +<h1 id="restore-from-backup">Restore from backup</h1> + +<p>At this point the scheduler is ready to rehydrate from the backup:</p> + +<ul> +<li><p>Identify the leading scheduler by:</p> + +<ul> +<li>examining the <code>scheduler_lifecycle_LEADER_AWAITING_REGISTRATION</code> metric at the scheduler +<code>/vars</code> endpoint. Leader will have 1. All other replicas - 0.</li> +<li>examining scheduler logs</li> +<li>or examining Zookeeper registration under the path defined by <code>-zk_endpoints</code> +and <code>-serverset_path</code></li> +</ul></li> +<li><p>Locate the desired backup file, copy it to the leading scheduler’s <code>-backup_dir</code> folder and stage +recovery by running the following command on a leader +<code>aurora_admin scheduler_stage_recovery --bypass-leader-redirect <cluster> scheduler-backup-<yyyy-MM-dd-HH-mm></code></p></li> +<li><p>At this point, the recovery snapshot is staged and available for manual verification/modification +via <code>aurora_admin scheduler_print_recovery_tasks --bypass-leader-redirect</code> and +<code>scheduler_delete_recovery_tasks --bypass-leader-redirect</code> commands. +See <code>aurora_admin help <command></code> for usage details.</p></li> +<li><p>Commit recovery. This instructs the scheduler to overwrite the existing Mesos replicated log with +the provided backup snapshot and initiate a mandatory failover +<code>aurora_admin scheduler_commit_recovery --bypass-leader-redirect <cluster></code></p></li> +</ul> + +<h1 id="cleanup">Cleanup</h1> + +<p>Undo any modification done during <a href="#preparation">Preparation</a> sequence.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
Added: aurora/site/publish/documentation/0.17.0/operations/configuration/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/operations/configuration/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/operations/configuration/index.html (added) +++ aurora/site/publish/documentation/0.17.0/operations/configuration/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,403 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/operations/configuration/'" + value="0.17.0"> + <option value="0.17.0" + selected="selected"> + 0.17.0 + (latest) + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="scheduler-configuration">Scheduler Configuration</h1> + +<p>The Aurora scheduler can take a variety of configuration options through command-line arguments. +Examples are available under <code>examples/scheduler/</code>. For a list of available Aurora flags and their +documentation, see <a href="../../reference/scheduler-configuration/">Scheduler Configuration Reference</a>.</p> + +<h2 id="a-note-on-configuration">A Note on Configuration</h2> + +<p>Like Mesos, Aurora uses command-line flags for runtime configuration. As such the Aurora +“configuration file” is typically a <code>scheduler.sh</code> shell script of the form.</p> +<pre class="highlight shell"><code><span style="color: #999988;font-style: italic">#!/bin/bash</span> +<span style="color: #008080">AURORA_HOME</span><span style="color: #000000;font-weight: bold">=</span>/usr/local/aurora-scheduler + +<span style="color: #999988;font-style: italic"># Flags controlling the JVM.</span> +<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=(</span> + -Xmx2g + -Xms2g + <span style="color: #999988;font-style: italic"># GC tuning, etc.</span> +<span style="color: #000000;font-weight: bold">)</span> + +<span style="color: #999988;font-style: italic"># Flags controlling the scheduler.</span> +<span style="color: #008080">AURORA_FLAGS</span><span style="color: #000000;font-weight: bold">=(</span> + <span style="color: #999988;font-style: italic"># Port for client RPCs and the web UI</span> + -http_port<span style="color: #000000;font-weight: bold">=</span>8081 + <span style="color: #999988;font-style: italic"># Log configuration, etc.</span> +<span style="color: #000000;font-weight: bold">)</span> + +<span style="color: #999988;font-style: italic"># Environment variables controlling libmesos</span> +<span style="color: #0086B3">export </span><span style="color: #008080">JAVA_HOME</span><span style="color: #000000;font-weight: bold">=</span>... +<span style="color: #0086B3">export </span><span style="color: #008080">GLOG_v</span><span style="color: #000000;font-weight: bold">=</span>1 +<span style="color: #999988;font-style: italic"># Port and public ip used to communicate with the Mesos master and for the replicated log</span> +<span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_PORT</span><span style="color: #000000;font-weight: bold">=</span>8083 +<span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_IP</span><span style="color: #000000;font-weight: bold">=</span>192.168.33.7 + +<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">JAVA_OPTS</span><span style="background-color: #f8f8f8">[*]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> <span style="color: #0086B3">exec</span> <span style="color: #d14">"</span><span style="color: #008080">$AURORA_HOME</span><span style="color: #d14">/bin/aurora-scheduler"</span> <span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">AURORA_FLAGS</span><span style="background-color: #f8f8f8">[@]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> +</code></pre> + +<p>That way Aurora’s current flags are visible in <code>ps</code> and in the <code>/vars</code> admin endpoint.</p> + +<h2 id="replicated-log-configuration">Replicated Log Configuration</h2> + +<p>Aurora schedulers use ZooKeeper to discover log replicas and elect a leader. Only one scheduler is +leader at a given time - the other schedulers follow log writes and prepare to take over as leader +but do not communicate with the Mesos master. Either 3 or 5 schedulers are recommended in a +production deployment depending on failure tolerance and they must have persistent storage.</p> + +<p>Below is a summary of scheduler storage configuration flags that either don’t have default values +or require attention before deploying in a production environment.</p> + +<h3 id="native_log_quorum_size"><code>-native_log_quorum_size</code></h3> + +<p>Defines the Mesos replicated log quorum size. In a cluster with <code>N</code> schedulers, the flag +<code>-native_log_quorum_size</code> should be set to <code>floor(N/2) + 1</code>. So in a cluster with 1 scheduler +it should be set to <code>1</code>, in a cluster with 3 it should be set to <code>2</code>, and in a cluster of 5 it +should be set to <code>3</code>.</p> + +<table><thead> +<tr> +<th>Number of schedulers (N)</th> +<th><code>-native_log_quorum_size</code> setting (<code>floor(N/2) + 1</code>)</th> +</tr> +</thead><tbody> +<tr> +<td>1</td> +<td>1</td> +</tr> +<tr> +<td>3</td> +<td>2</td> +</tr> +<tr> +<td>5</td> +<td>3</td> +</tr> +<tr> +<td>7</td> +<td>4</td> +</tr> +</tbody></table> + +<p><em>Incorrectly setting this flag will cause data corruption to occur!</em></p> + +<h3 id="native_log_file_path"><code>-native_log_file_path</code></h3> + +<p>Location of the Mesos replicated log files. Consider allocating a dedicated disk (preferably SSD) +for Mesos replicated log files to ensure optimal storage performance.</p> + +<h3 id="native_log_zk_group_path"><code>-native_log_zk_group_path</code></h3> + +<p>ZooKeeper path used for Mesos replicated log quorum discovery.</p> + +<p>See <a href="https://github.com/apache/aurora/blob/rel/0.17.0/src/main/java/org/apache/aurora/scheduler/log/mesos/MesosLogStreamModule.java";>code</a> for +other available Mesos replicated log configuration options and default values.</p> + +<h3 id="changing-the-quorum-size">Changing the Quorum Size</h3> + +<p>Special care needs to be taken when changing the size of the Aurora scheduler quorum. +Since Aurora uses a Mesos replicated log, similar steps need to be followed as when +<a href="http://mesos.apache.org/documentation/latest/operational-guide";>changing the Mesos quorum size</a>.</p> + +<p>As a preparation, increase <code>-native_log_quorum_size</code> on each existing scheduler and restart them. +When updating from 3 to 5 schedulers, the quorum size would grow from 2 to 3.</p> + +<p>When starting the new schedulers, use the <code>-native_log_quorum_size</code> set to the new value. Failing to +first increase the quorum size on running schedulers can in some cases result in corruption +or truncating of the replicated log used by Aurora. In that case, see the documentation on +<a href="../backup-restore/">recovering from backup</a>.</p> + +<h2 id="backup-configuration">Backup Configuration</h2> + +<p>Configuration options for the Aurora scheduler backup manager.</p> + +<ul> +<li><code>-backup_interval</code>: The interval on which the scheduler writes local storage backups. The default is every hour.</li> +<li><code>-backup_dir</code>: Directory to write backups to.</li> +<li><code>-max_saved_backups</code>: Maximum number of backups to retain before deleting the oldest backup(s).</li> +</ul> + +<h2 id="resource-isolation">Resource Isolation</h2> + +<p>For proper CPU, memory, and disk isolation as mentioned in our <a href="../../features/resource-isolation/">enduser documentation</a>, +we recommend to add the following isolators to the <code>--isolation</code> flag of the Mesos agent:</p> + +<ul> +<li><code>cgroups/cpu</code></li> +<li><code>cgroups/mem</code></li> +<li><code>disk/du</code></li> +</ul> + +<p>In addition, we recommend to set the following <a href="http://mesos.apache.org/documentation/latest/configuration/";>agent flags</a>:</p> + +<ul> +<li><code>--cgroups_limit_swap</code> to enable memory limits on both memory and swap instead of just memory. +Alternatively, you could disable swap on your agent hosts.</li> +<li><code>--cgroups_enable_cfs</code> to enable hard limits on CPU resources via the CFS bandwidth limiting +feature.</li> +<li><code>--enforce_container_disk_quota</code> to enable disk quota enforcement for containers.</li> +</ul> + +<p>To enable the optional GPU support in Mesos, please see the GPU related flags in the +<a href="http://mesos.apache.org/documentation/latest/configuration/";>Mesos configuration</a>. +To enable the corresponding feature in Aurora, you have to start the scheduler with the +flag</p> +<pre class="highlight plaintext"><code>-allow_gpu_resource=true +</code></pre> + +<p>If you want to use revocable resources, first follow the +<a href="http://mesos.apache.org/documentation/latest/oversubscription/";>Mesos oversubscription documentation</a> +and then set set this Aurora scheduler flag to allow receiving revocable Mesos offers:</p> +<pre class="highlight plaintext"><code>-receive_revocable_resources=true +</code></pre> + +<p>Both CPUs and RAM are supported as revocable resources. The former is enabled by the default, +the latter needs to be enabled via:</p> +<pre class="highlight plaintext"><code>-enable_revocable_ram=true +</code></pre> + +<p>Unless you want to use the <a href="https://github.com/apache/aurora/blob/rel/0.17.0/src/main/resources/org/apache/aurora/scheduler/tiers.json";>default</a> +tier configuration, you will also have to specify a file path:</p> +<pre class="highlight plaintext"><code>-tier_config=path/to/tiers/config.json +</code></pre> + +<h2 id="containers">Containers</h2> + +<p>Both the Mesos and Docker containerizers require configuration of the Mesos agent.</p> + +<h3 id="mesos-containerizer">Mesos Containerizer</h3> + +<p>The minimal agent configuration requires to enable Docker and Appc image support for the Mesos +containerizer:</p> +<pre class="highlight plaintext"><code>--containerizers=mesos +--image_providers=appc,docker +--isolation=filesystem/linux,docker/runtime # as an addition to your other isolators +</code></pre> + +<p>Further details can be found in the corresponding <a href="http://mesos.apache.org/documentation/latest/container-image/";>Mesos documentation</a>.</p> + +<h3 id="docker-containerizer">Docker Containerizer</h3> + +<p>The <a href="http://mesos.apache.org/documentation/latest/docker-containerizer/";>Docker containerizer</a> +requires the Docker engine is installed on each agent host. In addition, it must be enabled on the +Mesos agents by launching them with the option:</p> +<pre class="highlight plaintext"><code>--containerizers=mesos,docker +</code></pre> + +<p>If you would like to run a container with a read-only filesystem, it may also be necessary to use +the scheduler flag <code>-thermos_home_in_sandbox</code> in order to set HOME to the sandbox +before the executor runs. This will make sure that the executor/runner PEX extractions happens +inside of the sandbox instead of the container filesystem root.</p> + +<p>If you would like to supply your own parameters to <code>docker run</code> when launching jobs in docker +containers, you may use the following flags:</p> +<pre class="highlight plaintext"><code>-allow_docker_parameters +-default_docker_parameters +</code></pre> + +<p><code>-allow_docker_parameters</code> controls whether or not users may pass their own configuration parameters +through the job configuration files. If set to <code>false</code> (the default), the scheduler will reject +jobs with custom parameters. <em>NOTE</em>: this setting should be used with caution as it allows any job +owner to specify any parameters they wish, including those that may introduce security concerns +(<code>privileged=true</code>, for example).</p> + +<p><code>-default_docker_parameters</code> allows a cluster operator to specify a universal set of parameters that +should be used for every container that does not have parameters explicitly configured at the job +level. The argument accepts a multimap format:</p> +<pre class="highlight plaintext"><code>-default_docker_parameters="read-only=true,tmpfs=/tmp,tmpfs=/run" +</code></pre> + +<h3 id="common-options">Common Options</h3> + +<p>The following Aurora options work for both containerizers.</p> + +<p>A scheduler flag, <code>-global_container_mounts</code> allows mounting paths from the host (i.e the agent machine) +into all containers on that host. The format is a comma separated list of host<em>path:container</em>path[:mode] +tuples. For example <code>-global_container_mounts=/opt/secret_keys_dir:/mnt/secret_keys_dir:ro</code> mounts +<code>/opt/secret_keys_dir</code> from the agents into all launched containers. Valid modes are <code>ro</code> and <code>rw</code>.</p> + +<h2 id="thermos-process-logs">Thermos Process Logs</h2> + +<h3 id="log-destination">Log destination</h3> + +<p>By default, Thermos will write process stdout/stderr to log files in the sandbox. Process object +configuration allows specifying alternate log file destinations like streamed stdout/stderr or +suppression of all log output. Default behavior can be configured for the entire cluster with the +following flag (through the <code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p> +<pre class="highlight plaintext"><code>--runner-logger-destination=both +</code></pre> + +<p><code>both</code> configuration will send logs to files and stream to parent stdout/stderr outputs.</p> + +<p>See <a href="../../reference/configuration/#logger">Configuration Reference</a> for all destination options.</p> + +<h3 id="log-rotation">Log rotation</h3> + +<p>By default, Thermos will not rotate the stdout/stderr logs from child processes and they will grow +without bound. An individual user may change this behavior via configuration on the Process object, +but it may also be desirable to change the default configuration for the entire cluster. +In order to enable rotation by default, the following flags can be applied to Thermos (through the +<code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p> +<pre class="highlight plaintext"><code>--runner-logger-mode=rotate +--runner-rotate-log-size-mb=100 +--runner-rotate-log-backups=10 +</code></pre> + +<p>In the above example, each instance of the Thermos runner will rotate stderr/stdout logs once they +reach 100 MiB in size and keep a maximum of 10 backups. If a user has provided a custom setting for +their process, it will override these default settings.</p> + +<h2 id="thermos-executor-wrapper">Thermos Executor Wrapper</h2> + +<p>If you need to do computation before starting the Thermos executor (for example, setting a different +<code>--announcer-hostname</code> parameter for every executor), then the Thermos executor should be invoked +inside a wrapper script. In such a case, the aurora scheduler should be started with +<code>-thermos_executor_path</code> pointing to the wrapper script and <code>-thermos_executor_resources</code> set to a +comma separated string of all the resources that should be copied into the sandbox (including the +original Thermos executor). Ensure the wrapper script does not access resources outside of the +sandbox, as when the script is run from within a Docker container those resources may not exist.</p> + +<p>For example, to wrap the executor inside a simple wrapper, the scheduler will be started like this +<code>-thermos_executor_path=/path/to/wrapper.sh -thermos_executor_resources=/usr/share/aurora/bin/thermos_executor.pex</code></p> + +<h2 id="custom-executors">Custom Executors</h2> + +<p>The scheduler can be configured to utilize a custom executor by specifying the <code>-custom_executor_config</code> flag. +The flag must be set to the path of a valid executor configuration file.</p> + +<p>For more information on this feature please see the custom executors <a href="../../features/custom-executors/">documentation</a>.</p> + +<h2 id="a-note-on-increasing-executor-overhead">A note on increasing executor overhead</h2> + +<p>Increasing executor overhead on an existing cluster, whether it be for custom executors or for Thermos, +will result in degraded preemption performance until all task which began life with the previous +executor configuration with less overhead are preempted/restarted.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.17.0/operations/installation/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/operations/installation/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/operations/installation/index.html (added) +++ aurora/site/publish/documentation/0.17.0/operations/installation/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,455 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/operations/installation/'" + value="0.17.0"> + <option value="0.17.0" + selected="selected"> + 0.17.0 + (latest) + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="installing-aurora">Installing Aurora</h1> + +<p>Source and binary distributions can be found on our +<a href="https://aurora.apache.org/downloads/";>downloads</a> page. Installing from binary packages is +recommended for most.</p> + +<ul> +<li><a href="#installing-the-scheduler">Installing the scheduler</a></li> +<li><a href="#installing-worker-components">Installing worker components</a></li> +<li><a href="#installing-the-client">Installing the client</a></li> +<li><a href="#installing-mesos">Installing Mesos</a></li> +<li><a href="#troubleshooting">Troubleshooting</a></li> +</ul> + +<p>If our binay packages don’t suite you, our package build toolchain makes it easy to build your +own packages. See the <a href="https://github.com/apache/aurora-packaging";>instructions</a> to learn how.</p> + +<h2 id="machine-profiles">Machine profiles</h2> + +<p>Given that many of these components communicate over the network, there are numerous ways you could +assemble them to create an Aurora cluster. The simplest way is to think in terms of three machine +profiles:</p> + +<h3 id="coordinator">Coordinator</h3> + +<p><strong>Components</strong>: ZooKeeper, Aurora scheduler, Mesos master</p> + +<p>A small number of machines (typically 3 or 5) responsible for cluster orchestration. In most cases +it is fine to co-locate these components in anything but very large clusters (> 1000 machines). +Beyond that point, operators will likely want to manage these services on separate machines.</p> + +<p>In practice, 5 coordinators have been shown to reliably manage clusters with tens of thousands of +machines.</p> + +<h3 id="worker">Worker</h3> + +<p><strong>Components</strong>: Aurora executor, Aurora observer, Mesos agent</p> + +<p>The bulk of the cluster, where services will actually run.</p> + +<h3 id="client">Client</h3> + +<p><strong>Components</strong>: Aurora client, Aurora admin client</p> + +<p>Any machines that users submit jobs from.</p> + +<h2 id="installing-the-scheduler">Installing the scheduler</h2> + +<h3 id="ubuntu-trusty">Ubuntu Trusty</h3> + +<ol> +<li><p>Install Mesos +Skip down to <a href="#mesos-on-ubuntu-trusty">install mesos</a>, then run:</p> +<pre class="highlight plaintext"><code>sudo start mesos-master +</code></pre></li> +<li><p>Install ZooKeeper</p> +<pre class="highlight plaintext"><code>sudo apt-get install -y zookeeperd +</code></pre></li> +<li><p>Install the Aurora scheduler</p> +<pre class="highlight plaintext"><code>sudo add-apt-repository -y ppa:openjdk-r/ppa +sudo apt-get update +sudo apt-get install -y openjdk-8-jre-headless wget + +sudo update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java + +wget -c https://apache.bintray.com/aurora/ubuntu-trusty/aurora-scheduler_0.17.0_amd64.deb +sudo dpkg -i aurora-scheduler_0.17.0_amd64.deb +</code></pre></li> +</ol> + +<h3 id="centos-7">CentOS 7</h3> + +<ol> +<li><p>Install Mesos +Skip down to <a href="#mesos-on-centos-7">install mesos</a>, then run:</p> +<pre class="highlight plaintext"><code>sudo systemctl start mesos-master +</code></pre></li> +<li><p>Install ZooKeeper</p> +<pre class="highlight plaintext"><code>sudo rpm -Uvh https://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm +sudo yum install -y java-1.8.0-openjdk-headless zookeeper-server + +sudo service zookeeper-server init +sudo systemctl start zookeeper-server +</code></pre></li> +<li><p>Install the Aurora scheduler</p> +<pre class="highlight plaintext"><code>sudo yum install -y wget + +wget -c https://apache.bintray.com/aurora/centos-7/aurora-scheduler-0.17.0-1.el7.centos.aurora.x86_64.rpm +sudo yum install -y aurora-scheduler-0.17.0-1.el7.centos.aurora.x86_64.rpm +</code></pre></li> +</ol> + +<h3 id="finalizing">Finalizing</h3> + +<p>By default, the scheduler will start in an uninitialized mode. This is because external +coordination is necessary to be certain operator error does not result in a quorum of schedulers +starting up and believing their databases are empty when in fact they should be re-joining a +cluster.</p> + +<p>Because of this, a fresh install of the scheduler will need intervention to start up. First, +stop the scheduler service. +Ubuntu: <code>sudo stop aurora-scheduler</code> +CentOS: <code>sudo systemctl stop aurora</code></p> + +<p>Now initialize the database:</p> +<pre class="highlight plaintext"><code>sudo -u aurora mkdir -p /var/lib/aurora/scheduler/db +sudo -u aurora mesos-log initialize --path=/var/lib/aurora/scheduler/db +</code></pre> + +<p>Now you can start the scheduler back up. +Ubuntu: <code>sudo start aurora-scheduler</code> +CentOS: <code>sudo systemctl start aurora</code></p> + +<h2 id="installing-worker-components">Installing worker components</h2> + +<h3 id="ubuntu-trusty">Ubuntu Trusty</h3> + +<ol> +<li><p>Install Mesos +Skip down to <a href="#mesos-on-ubuntu-trusty">install mesos</a>, then run:</p> +<pre class="highlight plaintext"><code>start mesos-slave +</code></pre></li> +<li><p>Install Aurora executor and observer</p> +<pre class="highlight plaintext"><code>sudo apt-get install -y python2.7 wget + +# NOTE: This appears to be a missing dependency of the mesos deb package and is needed +# for the python mesos native bindings. +sudo apt-get -y install libcurl4-nss-dev + +wget -c https://apache.bintray.com/aurora/ubuntu-trusty/aurora-executor_0.17.0_amd64.deb +sudo dpkg -i aurora-executor_0.17.0_amd64.deb +</code></pre></li> +</ol> + +<h3 id="centos-7">CentOS 7</h3> + +<ol> +<li><p>Install Mesos +Skip down to <a href="#mesos-on-centos-7">install mesos</a>, then run:</p> +<pre class="highlight plaintext"><code>sudo systemctl start mesos-slave +</code></pre></li> +<li><p>Install Aurora executor and observer</p> +<pre class="highlight plaintext"><code>sudo yum install -y python2 wget + +wget -c https://apache.bintray.com/aurora/centos-7/aurora-executor-0.17.0-1.el7.centos.aurora.x86_64.rpm +sudo yum install -y aurora-executor-0.17.0-1.el7.centos.aurora.x86_64.rpm +</code></pre></li> +</ol> + +<h3 id="configuration">Configuration</h3> + +<p>The executor typically does not require configuration. Command line arguments can +be passed to the executor using a command line argument on the scheduler.</p> + +<p>The observer needs to be configured to look at the correct mesos directory in order to find task +sandboxes. You should 1st find the Mesos working directory by looking for the Mesos agent +<code>--work_dir</code> flag. You should see something like:</p> +<pre class="highlight plaintext"><code> ps -eocmd | grep "mesos-slave" | grep -v grep | tr ' ' '\n' | grep "\--work_dir" + --work_dir=/var/lib/mesos +</code></pre> + +<p>If the flag is not set, you can view the default value like so:</p> +<pre class="highlight plaintext"><code> mesos-slave --help + Usage: mesos-slave [options] + + ... + --work_dir=VALUE Directory path to place framework work directories + (default: /tmp/mesos) + ... +</code></pre> + +<p>The value you find for <code>--work_dir</code>, <code>/var/lib/mesos</code> in this example, should match the Aurora +observer value for <code>--mesos-root</code>. You can look for that setting in a similar way on a worker +node by grepping for <code>thermos_observer</code> and <code>--mesos-root</code>. If the flag is not set, you can view +the default value like so:</p> +<pre class="highlight plaintext"><code> thermos_observer -h + Options: + ... + --mesos-root=MESOS_ROOT + The mesos root directory to search for Thermos + executor sandboxes [default: /var/lib/mesos] + ... +</code></pre> + +<p>In this case the default is <code>/var/lib/mesos</code> and we have a match. If there is no match, you can +either adjust the mesos-master start script(s) and restart the master(s) or else adjust the +Aurora observer start scripts and restart the observers. To adjust the Aurora observer:</p> + +<h4 id="ubuntu-trusty">Ubuntu Trusty</h4> +<pre class="highlight plaintext"><code>sudo sh -c 'echo "MESOS_ROOT=/tmp/mesos" >> /etc/default/thermos' +</code></pre> + +<h4 id="centos-7">CentOS 7</h4> + +<p>Make an edit to add the <code>--mesos-root</code> flag resulting in something like:</p> +<pre class="highlight plaintext"><code>grep -A5 OBSERVER_ARGS /etc/sysconfig/thermos +OBSERVER_ARGS=( + --port=1338 + --mesos-root=/tmp/mesos + --log_to_disk=NONE + --log_to_stderr=google:INFO +) +</code></pre> + +<h2 id="installing-the-client">Installing the client</h2> + +<h3 id="ubuntu-trusty">Ubuntu Trusty</h3> +<pre class="highlight plaintext"><code>sudo apt-get install -y python2.7 wget + +wget -c https://apache.bintray.com/aurora/ubuntu-trusty/aurora-tools_0.17.0_amd64.deb +sudo dpkg -i aurora-tools_0.17.0_amd64.deb +</code></pre> + +<h3 id="centos-7">CentOS 7</h3> +<pre class="highlight plaintext"><code>sudo yum install -y python2 wget + +wget -c https://apache.bintray.com/aurora/centos-7/aurora-tools-0.17.0-1.el7.centos.aurora.x86_64.rpm +sudo yum install -y aurora-tools-0.17.0-1.el7.centos.aurora.x86_64.rpm +</code></pre> + +<h3 id="mac-os-x">Mac OS X</h3> +<pre class="highlight plaintext"><code>brew upgrade +brew install aurora-cli +</code></pre> + +<h3 id="configuration">Configuration</h3> + +<p>Client configuration lives in a json file that describes the clusters available and how to reach +them. By default this file is at <code>/etc/aurora/clusters.json</code>.</p> + +<p>Jobs may be submitted to the scheduler using the client, and are described with +<a href="../../reference/configuration/">job configurations</a> expressed in <code>.aurora</code> files. Typically you will +maintain a single job configuration file to describe one or more deployment environments (e.g. +dev, test, prod) for a production job.</p> + +<h2 id="installing-mesos">Installing Mesos</h2> + +<p>Mesos uses a single package for the Mesos master and agent. As a result, the package dependencies +are identical for both.</p> + +<h3 id="mesos-on-ubuntu-trusty">Mesos on Ubuntu Trusty</h3> +<pre class="highlight plaintext"><code>sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF +DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') +CODENAME=$(lsb_release -cs) + +echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" | \ + sudo tee /etc/apt/sources.list.d/mesosphere.list +sudo apt-get -y update + +# Use `apt-cache showpkg mesos | grep [version]` to find the exact version. +sudo apt-get -y install mesos=1.1.0-2.0.107.ubuntu1404_amd64.deb +</code></pre> + +<h3 id="mesos-on-centos-7">Mesos on CentOS 7</h3> +<pre class="highlight plaintext"><code>sudo rpm -Uvh https://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm +sudo yum -y install mesos-1.1.0 +</code></pre> + +<h2 id="troubleshooting">Troubleshooting</h2> + +<p>So you’ve started your first cluster and are running into some issues? We’ve collected some common +stumbling blocks and solutions here to help get you moving.</p> + +<h3 id="replicated-log-not-initialized">Replicated log not initialized</h3> + +<h4 id="symptoms">Symptoms</h4> + +<ul> +<li>Scheduler RPCs and web interface claim <code>Storage is not READY</code></li> +<li>Scheduler log repeatedly prints messages like</li> +</ul> +<pre class="highlight plaintext"><code> I1016 16:12:27.234133 26081 replica.cpp:638] Replica in EMPTY status + received a broadcasted recover request + I1016 16:12:27.234256 26084 recover.cpp:188] Received a recover response + from a replica in EMPTY status +</code></pre> + +<h4 id="solution">Solution</h4> + +<p>When you create a new cluster, you need to inform a quorum of schedulers that they are safe to +consider their database to be empty by <a href="#finalizing">initializing</a> the +replicated log. This is done to prevent the scheduler from modifying the cluster state in the event +of multiple simultaneous disk failures or, more likely, misconfiguration of the replicated log path.</p> + +<h3 id="scheduler-not-registered">Scheduler not registered</h3> + +<h4 id="symptoms">Symptoms</h4> + +<p>Scheduler log contains</p> +<pre class="highlight plaintext"><code>Framework has not been registered within the tolerated delay. +</code></pre> + +<h4 id="solution">Solution</h4> + +<p>Double-check that the scheduler is configured correctly to reach the Mesos master. If you are registering +the master in ZooKeeper, make sure command line argument to the master:</p> +<pre class="highlight plaintext"><code>--zk=zk://$ZK_HOST:2181/mesos/master +</code></pre> + +<p>is the same as the one on the scheduler:</p> +<pre class="highlight plaintext"><code>-mesos_master_address=zk://$ZK_HOST:2181/mesos/master +</code></pre> + +<h3 id="scheduler-not-running">Scheduler not running</h3> + +<h3 id="symptom">Symptom</h3> + +<p>The scheduler process commits suicide regularly. This happens under error conditions, but +also on purpose in regular intervals.</p> + +<h2 id="solution">Solution</h2> + +<p>Aurora is meant to be run under supervision. You have to configure a supervisor like +<a href="http://mmonit.com/monit/";>Monit</a> or <a href="http://supervisord.org/";>supervisord</a> to run the scheduler +and restart it whenever it fails or exists on purpose.</p> + +<p>Aurora supports an active health checking protocol on its admin HTTP interface - if a <code>GET /health</code> +times out or returns anything other than <code>200 OK</code> the scheduler process is unhealthy and should be +restarted.</p> + +<p>For example, monit can be configured with</p> +<pre class="highlight plaintext"><code>if failed port 8081 send "GET /health HTTP/1.0\r\n" expect "OK\n" with timeout 2 seconds for 10 cycles then restart +</code></pre> + +<p>assuming you set <code>-http_port=8081</code>.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.17.0/operations/monitoring/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/operations/monitoring/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/operations/monitoring/index.html (added) +++ aurora/site/publish/documentation/0.17.0/operations/monitoring/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,325 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/operations/monitoring/'" + value="0.17.0"> + <option value="0.17.0" + selected="selected"> + 0.17.0 + (latest) + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="monitoring-your-aurora-cluster">Monitoring your Aurora cluster</h1> + +<p>Before you start running important services in your Aurora cluster, it’s important to set up +monitoring and alerting of Aurora itself. Most of your monitoring can be against the scheduler, +since it will give you a global view of what’s going on.</p> + +<h2 id="reading-stats">Reading stats</h2> + +<p>The scheduler exposes a <em>lot</em> of instrumentation data via its HTTP interface. You can get a quick +peek at the first few of these in our vagrant image:</p> +<pre class="highlight plaintext"><code>$ vagrant ssh -c 'curl -s localhost:8081/vars | head' +async_tasks_completed 1004 +attribute_store_fetch_all_events 15 +attribute_store_fetch_all_events_per_sec 0.0 +attribute_store_fetch_all_nanos_per_event 0.0 +attribute_store_fetch_all_nanos_total 3048285 +attribute_store_fetch_all_nanos_total_per_sec 0.0 +attribute_store_fetch_one_events 3391 +attribute_store_fetch_one_events_per_sec 0.0 +attribute_store_fetch_one_nanos_per_event 0.0 +attribute_store_fetch_one_nanos_total 454690753 +</code></pre> + +<p>These values are served as <code>Content-Type: text/plain</code>, with each line containing a space-separated metric +name and value. Values may be integers, doubles, or strings (note: strings are static, others +may be dynamic).</p> + +<p>If your monitoring infrastructure prefers JSON, the scheduler exports that as well:</p> +<pre class="highlight plaintext"><code>$ vagrant ssh -c 'curl -s localhost:8081/vars.json | python -mjson.tool | head' +{ + "async_tasks_completed": 1009, + "attribute_store_fetch_all_events": 15, + "attribute_store_fetch_all_events_per_sec": 0.0, + "attribute_store_fetch_all_nanos_per_event": 0.0, + "attribute_store_fetch_all_nanos_total": 3048285, + "attribute_store_fetch_all_nanos_total_per_sec": 0.0, + "attribute_store_fetch_one_events": 3409, + "attribute_store_fetch_one_events_per_sec": 0.0, + "attribute_store_fetch_one_nanos_per_event": 0.0, +</code></pre> + +<p>This will be the same data as above, served with <code>Content-Type: application/json</code>.</p> + +<h2 id="viewing-live-stat-samples-on-the-scheduler">Viewing live stat samples on the scheduler</h2> + +<p>The scheduler uses the Twitter commons stats library, which keeps an internal time-series database +of exported variables - nearly everything in <code>/vars</code> is available for instant graphing. This is +useful for debugging, but is not a replacement for an external monitoring system.</p> + +<p>You can view these graphs on a scheduler at <code>/graphview</code>. It supports some composition and +aggregation of values, which can be invaluable when triaging a problem. For example, if you have +the scheduler running in vagrant, check out these links: +<a href="http://192.168.33.7:8081/graphview?query=jvm_uptime_secs";>simple graph</a> +<a href="http://192.168.33.7:8081/graphview?query=rate(scheduler_log_native_append_nanos_total)%2Frate(scheduler_log_native_append_events)%2F1e6">complex composition</a></p> + +<h3 id="counters-and-gauges">Counters and gauges</h3> + +<p>Among numeric stats, there are two fundamental types of stats exported: <em>counters</em> and <em>gauges</em>. +Counters are guaranteed to be monotonically-increasing for the lifetime of a process, while gauges +may decrease in value. Aurora uses counters to represent things like the number of times an event +has occurred, and gauges to capture things like the current length of a queue. Counters are a +natural fit for accurate composition into <a href="http://en.wikipedia.org/wiki/Rate_ratio";>rate ratios</a> +(useful for sample-resistant latency calculation), while gauges are not.</p> + +<h1 id="alerting">Alerting</h1> + +<h2 id="quickstart">Quickstart</h2> + +<p>If you are looking for just bare-minimum alerting to get something in place quickly, set up alerting +on <code>framework_registered</code> and <code>task_store_LOST</code>. These will give you a decent picture of overall +health.</p> + +<h2 id="a-note-on-thresholds">A note on thresholds</h2> + +<p>One of the most difficult things in monitoring is choosing alert thresholds. With many of these +stats, there is no value we can offer as a threshold that will be guaranteed to work for you. It +will depend on the size of your cluster, number of jobs, churn of tasks in the cluster, etc. We +recommend you start with a strict value after viewing a small amount of collected data, and then +adjust thresholds as you see fit. Feel free to ask us if you would like to validate that your alerts +and thresholds make sense.</p> + +<h2 id="important-stats">Important stats</h2> + +<h3 id="jvm_uptime_secs"><code>jvm_uptime_secs</code></h3> + +<p>Type: integer counter</p> + +<p>The number of seconds the JVM process has been running. Comes from +<a href="http://docs.oracle.com/javase/7/docs/api/java/lang/management/RuntimeMXBean.html#getUptime()">RuntimeMXBean#getUptime()</a></p> + +<p>Detecting resets (decreasing values) on this stat will tell you that the scheduler is failing to +stay alive.</p> + +<p>Look at the scheduler logs to identify the reason the scheduler is exiting.</p> + +<h3 id="system_load_avg"><code>system_load_avg</code></h3> + +<p>Type: double gauge</p> + +<p>The current load average of the system for the last minute. Comes from +<a href="http://docs.oracle.com/javase/7/docs/api/java/lang/management/OperatingSystemMXBean.html?is-external=true#getSystemLoadAverage()">OperatingSystemMXBean#getSystemLoadAverage()</a>.</p> + +<p>A high sustained value suggests that the scheduler machine may be over-utilized.</p> + +<p>Use standard unix tools like <code>top</code> and <code>ps</code> to track down the offending process(es).</p> + +<h3 id="process_cpu_cores_utilized"><code>process_cpu_cores_utilized</code></h3> + +<p>Type: double gauge</p> + +<p>The current number of CPU cores in use by the JVM process. This should not exceed the number of +logical CPU cores on the machine. Derived from +<a href="http://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html";>OperatingSystemMXBean#getProcessCpuTime()</a></p> + +<p>A high sustained value indicates that the scheduler is overworked. Due to current internal design +limitations, if this value is sustained at <code>1</code>, there is a good chance the scheduler is under water.</p> + +<p>There are two main inputs that tend to drive this figure: task scheduling attempts and status +updates from Mesos. You may see activity in the scheduler logs to give an indication of where +time is being spent. Beyond that, it really takes good familiarity with the code to effectively +triage this. We suggest engaging with an Aurora developer.</p> + +<h3 id="task_store_lost"><code>task_store_LOST</code></h3> + +<p>Type: integer gauge</p> + +<p>The number of tasks stored in the scheduler that are in the <code>LOST</code> state, and have been rescheduled.</p> + +<p>If this value is increasing at a high rate, it is a sign of trouble.</p> + +<p>There are many sources of <code>LOST</code> tasks in Mesos: the scheduler, master, agent, and executor can all +trigger this. The first step is to look in the scheduler logs for <code>LOST</code> to identify where the +state changes are originating.</p> + +<h3 id="scheduler_resource_offers"><code>scheduler_resource_offers</code></h3> + +<p>Type: integer counter</p> + +<p>The number of resource offers that the scheduler has received.</p> + +<p>For a healthy scheduler, this value must be increasing over time.</p> + +<p>Assuming the scheduler is up and otherwise healthy, you will want to check if the master thinks it +is sending offers. You should also look at the master’s web interface to see if it has a large +number of outstanding offers that it is waiting to be returned.</p> + +<h3 id="framework_registered"><code>framework_registered</code></h3> + +<p>Type: binary integer counter</p> + +<p>Will be <code>1</code> for the leading scheduler that is registered with the Mesos master, <code>0</code> for passive +schedulers,</p> + +<p>A sustained period without a <code>1</code> (or where <code>sum() != 1</code>) warrants investigation.</p> + +<p>If there is no leading scheduler, look in the scheduler and master logs for why. If there are +multiple schedulers claiming leadership, this suggests a split brain and warrants filing a critical +bug.</p> + +<h3 id="rate-scheduler_log_native_append_nanos_total-rate-scheduler_log_native_append_events"><code>rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)</code></h3> + +<p>Type: rate ratio of integer counters</p> + +<p>This composes two counters to compute a windowed figure for the latency of replicated log writes.</p> + +<p>A hike in this value suggests disk bandwidth contention.</p> + +<p>Look in scheduler logs for any reported oddness with saving to the replicated log. Also use +standard tools like <code>vmstat</code> and <code>iotop</code> to identify whether the disk has become slow or +over-utilized. We suggest using a dedicated disk for the replicated log to mitigate this.</p> + +<h3 id="timed_out_tasks"><code>timed_out_tasks</code></h3> + +<p>Type: integer counter</p> + +<p>Tracks the number of times the scheduler has given up while waiting +(for <code>-transient_task_state_timeout</code>) to hear back about a task that is in a transient state +(e.g. <code>ASSIGNED</code>, <code>KILLING</code>), and has moved to <code>LOST</code> before rescheduling.</p> + +<p>This value is currently known to increase occasionally when the scheduler fails over +(<a href="https://issues.apache.org/jira/browse/AURORA-740";>AURORA-740</a>). However, any large spike in this +value warrants investigation.</p> + +<p>The scheduler will log when it times out a task. You should trace the task ID of the timed out +task into the master, agent, and/or executors to determine where the message was dropped.</p> + +<h3 id="http_500_responses_events"><code>http_500_responses_events</code></h3> + +<p>Type: integer counter</p> + +<p>The total number of HTTP 500 status responses sent by the scheduler. Includes API and asset serving.</p> + +<p>An increase warrants investigation.</p> + +<p>Look in scheduler logs to identify why the scheduler returned a 500, there should be a stack trace.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
