Added: aurora/site/publish/documentation/0.20.0/getting-started/vagrant/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/getting-started/vagrant/index.html?rev=1828294&view=auto ============================================================================== --- aurora/site/publish/documentation/0.20.0/getting-started/vagrant/index.html (added) +++ aurora/site/publish/documentation/0.20.0/getting-started/vagrant/index.html Tue Apr 3 23:54:44 2018 @@ -0,0 +1,295 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/getting-started/vagrant/'" + value="0.20.0"> + <option value="0.20.0" + selected="selected"> + 0.20.0 + (latest) + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="a-local-cluster-with-vagrant">A local Cluster with Vagrant</h1> + +<p>This document shows you how to configure a complete cluster using a virtual machine. This setup +replicates a real cluster in your development machine as closely as possible. After you complete +the steps outlined here, you will be ready to create and run your first Aurora job.</p> + +<p>The following sections describe these steps in detail:</p> + +<ol> +<li><a href="#overview">Overview</a></li> +<li><a href="#install-virtualbox-and-vagrant">Install VirtualBox and Vagrant</a></li> +<li><a href="#clone-the-aurora-repository">Clone the Aurora repository</a></li> +<li><a href="#start-the-local-cluster">Start the local cluster</a></li> +<li><a href="#log-onto-the-vm">Log onto the VM</a></li> +<li><a href="#run-your-first-job">Run your first job</a></li> +<li><a href="#rebuild-components">Rebuild components</a></li> +<li><a href="#shut-down-or-delete-your-local-cluster">Shut down or delete your local cluster</a></li> +<li><a href="#troubleshooting">Troubleshooting</a></li> +</ol> + +<h2 id="overview">Overview</h2> + +<p>The Aurora distribution includes a set of scripts that enable you to create a local cluster in +your development machine. These scripts use <a href="https://www.vagrantup.com/";>Vagrant</a> and +<a href="https://www.virtualbox.org/";>VirtualBox</a> to run and configure a virtual machine. Once the +virtual machine is running, the scripts install and initialize Aurora and any required components +to create the local cluster.</p> + +<h2 id="install-virtualbox-and-vagrant">Install VirtualBox and Vagrant</h2> + +<p>First, download and install <a href="https://www.virtualbox.org/";>VirtualBox</a> on your development machine.</p> + +<p>Then download and install <a href="https://www.vagrantup.com/";>Vagrant</a>. To verify that the installation +was successful, open a terminal window and type the <code>vagrant</code> command. You should see a list of +common commands for this tool.</p> + +<h2 id="clone-the-aurora-repository">Clone the Aurora repository</h2> + +<p>To obtain the Aurora source distribution, clone its Git repository using the following command:</p> +<pre class="highlight plaintext"><code> git clone git://git.apache.org/aurora.git +</code></pre> + +<h2 id="start-the-local-cluster">Start the local cluster</h2> + +<p>Now change into the <code>aurora/</code> directory, which contains the Aurora source code and +other scripts and tools:</p> +<pre class="highlight plaintext"><code> cd aurora/ +</code></pre> + +<p>To start the local cluster, type the following command:</p> +<pre class="highlight plaintext"><code> vagrant up +</code></pre> + +<p>This command uses the configuration scripts in the Aurora distribution to:</p> + +<ul> +<li>Download a Linux system image.</li> +<li>Start a virtual machine (VM) and configure it.</li> +<li>Install the required build tools on the VM.</li> +<li>Install Aurora’s requirements (like <a href="http://mesos.apache.org/";>Mesos</a> and +<a href="http://zookeeper.apache.org/";>Zookeeper</a>) on the VM.</li> +<li>Build and install Aurora from source on the VM.</li> +<li>Start Aurora’s services on the VM.</li> +</ul> + +<p>This process takes several minutes to complete.</p> + +<p>You may notice a warning that guest additions in the VM don’t match your version of VirtualBox. +This should generally be harmless, but you may wish to install a vagrant plugin to take care of +mismatches like this for you:</p> +<pre class="highlight plaintext"><code> vagrant plugin install vagrant-vbguest +</code></pre> + +<p>With this plugin installed, whenever you <code>vagrant up</code> the plugin will upgrade the guest additions +for you when a version mis-match is detected. You can read more about the plugin +<a href="https://github.com/dotless-de/vagrant-vbguest";>here</a>.</p> + +<p>To verify that Aurora is running on the cluster, visit the following URLs:</p> + +<ul> +<li>Scheduler - <a href="http://192.168.33.7:8081";>http://192.168.33.7:8081</a></li> +<li>Observer - <a href="http://192.168.33.7:1338";>http://192.168.33.7:1338</a></li> +<li>Mesos Master - <a href="http://192.168.33.7:5050";>http://192.168.33.7:5050</a></li> +<li>Mesos Agent - <a href="http://192.168.33.7:5051";>http://192.168.33.7:5051</a></li> +</ul> + +<h2 id="log-onto-the-vm">Log onto the VM</h2> + +<p>To SSH into the VM, run the following command in your development machine:</p> +<pre class="highlight plaintext"><code> vagrant ssh +</code></pre> + +<p>To verify that Aurora is installed in the VM, type the <code>aurora</code> command. You should see a list +of arguments and possible commands.</p> + +<p>The <code>/vagrant</code> directory on the VM is mapped to the <code>aurora/</code> local directory +from which you started the cluster. You can edit files inside this directory in your development +machine and access them from the VM under <code>/vagrant</code>.</p> + +<p>A pre-installed <code>clusters.json</code> file refers to your local cluster as <code>devcluster</code>, which you +will use in client commands.</p> + +<h2 id="run-your-first-job">Run your first job</h2> + +<p>Now that your cluster is up and running, you are ready to define and run your first job in Aurora. +For more information, see the <a href="../tutorial/">Aurora Tutorial</a>.</p> + +<h2 id="rebuild-components">Rebuild components</h2> + +<p>If you are changing Aurora code and would like to rebuild a component, you can use the <code>aurorabuild</code> +command on the VM to build and restart a component. This is considerably faster than destroying +and rebuilding your VM.</p> + +<p><code>aurorabuild</code> accepts a list of components to build and update. To get a list of supported +components, invoke the <code>aurorabuild</code> command with no arguments:</p> +<pre class="highlight plaintext"><code> vagrant ssh -c 'aurorabuild client' +</code></pre> + +<h2 id="shut-down-or-delete-your-local-cluster">Shut down or delete your local cluster</h2> + +<p>To shut down your local cluster, run the <code>vagrant halt</code> command in your development machine. To +start it again, run the <code>vagrant up</code> command.</p> + +<p>Once you are finished with your local cluster, or if you would otherwise like to start from scratch, +you can use the command <code>vagrant destroy</code> to turn off and delete the virtual file system.</p> + +<h2 id="troubleshooting">Troubleshooting</h2> + +<p>Most of the Vagrant related problems can be fixed by the following steps:</p> + +<ul> +<li>Destroying the vagrant environment with <code>vagrant destroy</code></li> +<li>Killing any orphaned VMs (see AURORA-499) with <code>virtualbox</code> UI or <code>VBoxManage</code> command line tool</li> +<li>Cleaning the repository of build artifacts and other intermediate output with <code>git clean -fdx</code></li> +<li>Bringing up the vagrant environment with <code>vagrant up</code></li> +</ul> + +<p>If that still doesn’t solve your problem, make sure to inspect the log files:</p> + +<ul> +<li>Scheduler: <code>/var/log/aurora/scheduler.log</code> or <code>sudo journalctl -u aurora-scheduler</code></li> +<li>Observer: <code>/var/log/thermos/observer.log</code> or <code>sudo journalctl -u thermos-observer</code></li> +<li>Mesos Master: <code>/var/log/mesos/mesos-master.INFO</code> (also see <code>.WARNING</code> and <code>.ERROR</code>)</li> +<li>Mesos Agent: <code>/var/log/mesos/mesos-slave.INFO</code> (also see <code>.WARNING</code> and <code>.ERROR</code>)</li> +</ul> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
Added: aurora/site/publish/documentation/0.20.0/images/CPUavailability.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/CPUavailability.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/CPUavailability.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/CompletedTasks.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/CompletedTasks.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/CompletedTasks.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/HelloWorldJob.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/HelloWorldJob.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/HelloWorldJob.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/RoleJobs.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/RoleJobs.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/RoleJobs.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/RunningJob.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/RunningJob.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/RunningJob.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/ScheduledJobs.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/ScheduledJobs.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/ScheduledJobs.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/TaskBreakdown.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/TaskBreakdown.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/TaskBreakdown.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/aurora_hierarchy.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/aurora_hierarchy.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/aurora_hierarchy.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/aurora_logo.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/aurora_logo.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/aurora_logo.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/components.odg URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/components.odg?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/components.odg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/components.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/components.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/components.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/debug-client-test.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/debug-client-test.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/debug-client-test.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/debugging-client-test.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/debugging-client-test.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/debugging-client-test.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/killedtask.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/killedtask.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/killedtask.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/lifeofatask.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/lifeofatask.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/lifeofatask.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/02_28_2015_apache_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/02_28_2015_apache_aurora_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/02_28_2015_apache_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/08_21_2014_past_present_future_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/08_21_2014_past_present_future_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/08_21_2014_past_present_future_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/runningtask.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/runningtask.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/runningtask.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/stderr.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/stderr.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/stderr.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/stdout.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/stdout.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/stdout.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/images/storage_hierarchy.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/images/storage_hierarchy.png?rev=1828294&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.20.0/images/storage_hierarchy.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/publish/documentation/0.20.0/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/index.html?rev=1828294&view=auto ============================================================================== --- aurora/site/publish/documentation/0.20.0/index.html (added) +++ aurora/site/publish/documentation/0.20.0/index.html Tue Apr 3 23:54:44 2018 @@ -0,0 +1,253 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/'" + value="0.20.0"> + <option value="0.20.0" + selected="selected"> + 0.20.0 + (latest) + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h2 id="introduction">Introduction</h2> + +<p>Apache Aurora is a service scheduler that runs on top of Apache Mesos, enabling you to run +long-running services, cron jobs, and ad-hoc jobs that take advantage of Apache Mesos’ scalability, +fault-tolerance, and resource isolation.</p> + +<p>We encourage you to ask questions on the <a href="http://aurora.apache.org/community/";>Aurora user list</a> or +the <code>#aurora</code> IRC channel on <code>irc.freenode.net</code>.</p> + +<h2 id="getting-started">Getting Started</h2> + +<p>Information for everyone new to Apache Aurora.</p> + +<ul> +<li><a href="getting-started/overview/">Aurora System Overview</a></li> +<li><a href="getting-started/tutorial/">Hello World Tutorial</a></li> +<li><a href="getting-started/vagrant/">Local cluster with Vagrant</a></li> +</ul> + +<h2 id="features">Features</h2> + +<p>Description of important Aurora features.</p> + +<ul> +<li><a href="features/containers/">Containers</a></li> +<li><a href="features/cron-jobs/">Cron Jobs</a></li> +<li><a href="features/custom-executors/">Custom Executors</a></li> +<li><a href="features/job-updates/">Job Updates</a></li> +<li><a href="features/multitenancy/">Multitenancy</a></li> +<li><a href="features/resource-isolation/">Resource Isolation</a></li> +<li><a href="features/constraints/">Scheduling Constraints</a></li> +<li><a href="features/services/">Services</a></li> +<li><a href="features/service-discovery/">Service Discovery</a></li> +<li><a href="features/sla-metrics/">SLA Metrics</a></li> +<li><a href="features/webhooks/">Webhooks</a></li> +</ul> + +<h2 id="operators">Operators</h2> + +<p>For those that wish to manage and fine-tune an Aurora cluster.</p> + +<ul> +<li><a href="operations/installation/">Installation</a></li> +<li><a href="operations/configuration/">Configuration</a></li> +<li><a href="operations/upgrades/">Upgrades</a></li> +<li><a href="operations/troubleshooting/">Troubleshooting</a></li> +<li><a href="operations/monitoring/">Monitoring</a></li> +<li><a href="operations/security/">Security</a></li> +<li><a href="operations/storage/">Storage</a></li> +<li><a href="operations/backup-restore/">Backup</a></li> +</ul> + +<h2 id="reference">Reference</h2> + +<p>The complete reference of commands, configuration options, and scheduler internals.</p> + +<ul> +<li><a href="reference/task-lifecycle/">Task lifecycle</a></li> +<li>Configuration (<code>.aurora</code> files) + +<ul> +<li><a href="reference/configuration/">Configuration Reference</a></li> +<li><a href="reference/configuration-tutorial/">Configuration Tutorial</a></li> +<li><a href="reference/configuration-best-practices/">Configuration Best Practices</a></li> +<li><a href="reference/configuration-templating/">Configuration Templating</a></li> +</ul></li> +<li>Aurora Client + +<ul> +<li><a href="reference/client-commands/">Client Commands</a></li> +<li><a href="reference/client-hooks/">Client Hooks</a></li> +<li><a href="reference/client-cluster-configuration/">Client Cluster Configuration</a></li> +</ul></li> +<li><a href="reference/scheduler-configuration/">Scheduler Configuration</a></li> +<li><a href="reference/observer-configuration/">Observer Configuration</a></li> +<li><a href="reference/scheduler-endpoints/">Endpoints</a></li> +</ul> + +<h2 id="additional-resources">Additional Resources</h2> + +<ul> +<li><a href="additional-resources/tools/">Tools integrating with Aurora</a></li> +<li><a href="additional-resources/presentations/">Presentation videos and slides</a></li> +</ul> + +<h2 id="developers">Developers</h2> + +<p>All the information you need to start modifying Aurora and contributing back to the project.</p> + +<ul> +<li><a href="contributing/">Contributing to the project</a></li> +<li><a href="development/committers-guide/">Committer’s Guide</a></li> +<li><a href="development/design-documents/">Design Documents</a></li> +<li>Developing the Aurora components: + +<ul> +<li><a href="development/client/">Client</a></li> +<li><a href="development/scheduler/">Scheduler</a></li> +<li><a href="development/ui/">Scheduler UI</a></li> +<li><a href="development/thermos/">Thermos</a></li> +<li><a href="development/thrift/">Thrift structures</a></li> +</ul></li> +</ul> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.20.0/operations/backup-restore/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/operations/backup-restore/index.html?rev=1828294&view=auto ============================================================================== --- aurora/site/publish/documentation/0.20.0/operations/backup-restore/index.html (added) +++ aurora/site/publish/documentation/0.20.0/operations/backup-restore/index.html Tue Apr 3 23:54:44 2018 @@ -0,0 +1,250 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/operations/backup-restore/'" + value="0.20.0"> + <option value="0.20.0" + selected="selected"> + 0.20.0 + (latest) + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="recovering-from-a-scheduler-backup">Recovering from a Scheduler Backup</h1> + +<p><strong>Be sure to read the entire page before attempting to restore from a backup, as it may have +unintended consequences.</strong></p> + +<h2 id="summary">Summary</h2> + +<p>The restoration procedure replaces the existing (possibly corrupted) Mesos replicated log with an +earlier, backed up, version and requires all schedulers to be taken down temporarily while +restoring. Once completed, the scheduler state resets to what it was when the backup was created. +This means any jobs/tasks created or updated after the backup are unknown to the scheduler and will +be killed shortly after the cluster restarts. All other tasks continue operating as normal.</p> + +<p>Usually, it is a bad idea to restore a backup that is not extremely recent (i.e. older than a few +hours). This is because the scheduler will expect the cluster to look exactly as the backup does, +so any tasks that have been rescheduled since the backup was taken will be killed.</p> + +<p>Instructions below have been verified in <a href="../../getting-started/vagrant/">Vagrant environment</a> and with minor +syntax/path changes should be applicable to any Aurora cluster.</p> + +<h2 id="preparation">Preparation</h2> + +<p>Follow these steps to prepare the cluster for restoring from a backup:</p> + +<ul> +<li><p>Stop all scheduler instances</p></li> +<li><p>Consider blocking external traffic on a port defined in <code>-http_port</code> for all schedulers to +prevent users from interacting with the scheduler during the restoration process. This will help +troubleshooting by reducing the scheduler log noise and prevent users from making changes that will +be erased after the backup snapshot is restored.</p></li> +<li><p>Configure <code>aurora_admin</code> access to run all commands listed in +<a href="#restore-from-backup">Restore from backup</a> section locally on the leading scheduler:</p> + +<ul> +<li>Make sure the <a href="../../reference/client-cluster-configuration/">clusters.json</a> file configured to +access scheduler directly. Set <code>scheduler_uri</code> setting and remove <code>zk</code>. Since leader can get +re-elected during the restore steps, consider doing it on all scheduler replicas.</li> +<li><p>Depending on your particular security approach you will need to either turn off scheduler +authorization by removing scheduler <code>-http_authentication_mechanism</code> flag or make sure the +direct scheduler access is properly authorized. E.g.: in case of Kerberos you will need to make +a <code>/etc/hosts</code> file change to match your local IP to the scheduler URL configured in keytabs:</p> + +<p><local_ip> <scheduler_domain_in_keytabs></p></li> +</ul></li> +<li><p>Next steps are required to put scheduler into a partially disabled state where it would still be +able to accept storage recovery requests but unable to schedule or change task states. This may be +accomplished by updating the following scheduler configuration options:</p> + +<ul> +<li>Set <code>-mesos_master_address</code> to a non-existent zk address. This will prevent scheduler from +registering with Mesos. E.g.: <code>-mesos_master_address=zk://localhost:1111/mesos/master</code></li> +<li><code>-max_registration_delay</code> - set to sufficiently long interval to prevent registration timeout +and as a result scheduler suicide. E.g: <code>-max_registration_delay=360mins</code></li> +<li>Make sure <code>-reconciliation_initial_delay</code> option is set high enough (e.g.: <code>365days</code>) to +prevent accidental task GC. This is important as scheduler will attempt to reconcile the cluster +state and will kill all tasks when restarted with an empty Mesos replicated log.</li> +</ul></li> +<li><p>Restart all schedulers</p></li> +</ul> + +<h2 id="cleanup-and-re-initialize-mesos-replicated-log">Cleanup and re-initialize Mesos replicated log</h2> + +<p>Get rid of the corrupted files and re-initialize Mesos replicated log:</p> + +<ul> +<li>Stop schedulers</li> +<li>Delete all files under <code>-native_log_file_path</code> on all schedulers</li> +<li>Initialize Mesos replica’s log file: <code>sudo mesos-log initialize --path=<-native_log_file_path></code></li> +<li>Start schedulers</li> +</ul> + +<h2 id="restore-from-backup">Restore from backup</h2> + +<p>At this point the scheduler is ready to rehydrate from the backup:</p> + +<ul> +<li><p>Identify the leading scheduler by:</p> + +<ul> +<li>examining the <code>scheduler_lifecycle_LEADER_AWAITING_REGISTRATION</code> metric at the scheduler +<code>/vars</code> endpoint. Leader will have 1. All other replicas - 0.</li> +<li>examining scheduler logs</li> +<li>or examining Zookeeper registration under the path defined by <code>-zk_endpoints</code> +and <code>-serverset_path</code></li> +</ul></li> +<li><p>Locate the desired backup file, copy it to the leading scheduler’s <code>-backup_dir</code> folder and stage +recovery by running the following command on a leader +<code>aurora_admin scheduler_stage_recovery --bypass-leader-redirect <cluster> scheduler-backup-<yyyy-MM-dd-HH-mm></code></p></li> +<li><p>At this point, the recovery snapshot is staged and available for manual verification/modification +via <code>aurora_admin scheduler_print_recovery_tasks --bypass-leader-redirect</code> and +<code>scheduler_delete_recovery_tasks --bypass-leader-redirect</code> commands. +See <code>aurora_admin help <command></code> for usage details.</p></li> +<li><p>Commit recovery. This instructs the scheduler to overwrite the existing Mesos replicated log with +the provided backup snapshot and initiate a mandatory failover +<code>aurora_admin scheduler_commit_recovery --bypass-leader-redirect <cluster></code></p></li> +</ul> + +<h2 id="cleanup">Cleanup</h2> + +<p>Undo any modification done during <a href="#preparation">Preparation</a> sequence.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.20.0/operations/configuration/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.20.0/operations/configuration/index.html?rev=1828294&view=auto ============================================================================== --- aurora/site/publish/documentation/0.20.0/operations/configuration/index.html (added) +++ aurora/site/publish/documentation/0.20.0/operations/configuration/index.html Tue Apr 3 23:54:44 2018 @@ -0,0 +1,513 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/operations/configuration/'" + value="0.20.0"> + <option value="0.20.0" + selected="selected"> + 0.20.0 + (latest) + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="scheduler-configuration">Scheduler Configuration</h1> + +<p>The Aurora scheduler can take a variety of configuration options through command-line arguments. +Examples are available under <code>examples/scheduler/</code>. For a list of available Aurora flags and their +documentation, see <a href="../../reference/scheduler-configuration/">Scheduler Configuration Reference</a>.</p> + +<h2 id="a-note-on-configuration">A Note on Configuration</h2> + +<p>Like Mesos, Aurora uses command-line flags for runtime configuration. As such the Aurora +“configuration file” is typically a <code>scheduler.sh</code> shell script of the form.</p> +<pre class="highlight shell"><code><span style="color: #999988;font-style: italic">#!/bin/bash</span> +<span style="color: #008080">AURORA_HOME</span><span style="color: #000000;font-weight: bold">=</span>/usr/local/aurora-scheduler + +<span style="color: #999988;font-style: italic"># Flags controlling the JVM.</span> +<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=(</span> + -Xmx2g + -Xms2g + <span style="color: #999988;font-style: italic"># GC tuning, etc.</span> +<span style="color: #000000;font-weight: bold">)</span> + +<span style="color: #999988;font-style: italic"># Flags controlling the scheduler.</span> +<span style="color: #008080">AURORA_FLAGS</span><span style="color: #000000;font-weight: bold">=(</span> + <span style="color: #999988;font-style: italic"># Port for client RPCs and the web UI</span> + -http_port<span style="color: #000000;font-weight: bold">=</span>8081 + <span style="color: #999988;font-style: italic"># Log configuration, etc.</span> +<span style="color: #000000;font-weight: bold">)</span> + +<span style="color: #999988;font-style: italic"># Environment variables controlling libmesos</span> +<span style="color: #0086B3">export </span><span style="color: #008080">JAVA_HOME</span><span style="color: #000000;font-weight: bold">=</span>... +<span style="color: #0086B3">export </span><span style="color: #008080">GLOG_v</span><span style="color: #000000;font-weight: bold">=</span>1 +<span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_PORT</span><span style="color: #000000;font-weight: bold">=</span>8083 +<span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_IP</span><span style="color: #000000;font-weight: bold">=</span>192.168.33.7 + +<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">JAVA_OPTS</span><span style="background-color: #f8f8f8">[*]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> <span style="color: #0086B3">exec</span> <span style="color: #d14">"</span><span style="color: #008080">$AURORA_HOME</span><span style="color: #d14">/bin/aurora-scheduler"</span> <span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">AURORA_FLAGS</span><span style="background-color: #f8f8f8">[@]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> +</code></pre> + +<p>That way Aurora’s current flags are visible in <code>ps</code> and in the <code>/vars</code> admin endpoint.</p> + +<h2 id="jvm-configuration">JVM Configuration</h2> + +<p>JVM settings are dependent on your environment and cluster size. They might require +custom tuning. As a starting point, we recommend:</p> + +<ul> +<li>Ensure the initial (<code>-Xms</code>) and maximum (<code>-Xmx</code>) heap size are idential to prevent heap resizing +at runtime.</li> +<li>Either <code>-XX:+UseConcMarkSweepGC</code> or <code>-XX:+UseG1GC -XX:+UseStringDeduplication</code> are +sane defaults for the garbage collector.</li> +<li><code>-Djava.net.preferIPv4Stack=true</code> makes sense in most cases as well.</li> +</ul> + +<h2 id="network-configuration">Network Configuration</h2> + +<p>By default, Aurora binds to all interfaces and auto-discovers its hostname. To reduce ambiguity +it helps to hardcode them though:</p> +<pre class="highlight plaintext"><code>-http_port=8081 +-ip=192.168.33.7 +-hostname="aurora1.us-east1.example.org" +</code></pre> + +<p>Two environment variables control the ip and port for the communication with the Mesos master +and for the replicated log used by Aurora:</p> +<pre class="highlight plaintext"><code>export LIBPROCESS_PORT=8083 +export LIBPROCESS_IP=192.168.33.7 +</code></pre> + +<p>It is important that those can be reached from all Mesos master and Aurora scheduler instances.</p> + +<h2 id="replicated-log-configuration">Replicated Log Configuration</h2> + +<p>Aurora schedulers use ZooKeeper to discover log replicas and elect a leader. Only one scheduler is +leader at a given time - the other schedulers follow log writes and prepare to take over as leader +but do not communicate with the Mesos master. Either 3 or 5 schedulers are recommended in a +production deployment depending on failure tolerance and they must have persistent storage.</p> + +<p>Below is a summary of scheduler storage configuration flags that either don’t have default values +or require attention before deploying in a production environment.</p> + +<h3 id="native_log_quorum_size"><code>-native_log_quorum_size</code></h3> + +<p>Defines the Mesos replicated log quorum size. In a cluster with <code>N</code> schedulers, the flag +<code>-native_log_quorum_size</code> should be set to <code>floor(N/2) + 1</code>. So in a cluster with 1 scheduler +it should be set to <code>1</code>, in a cluster with 3 it should be set to <code>2</code>, and in a cluster of 5 it +should be set to <code>3</code>.</p> + +<table><thead> +<tr> +<th>Number of schedulers (N)</th> +<th><code>-native_log_quorum_size</code> setting (<code>floor(N/2) + 1</code>)</th> +</tr> +</thead><tbody> +<tr> +<td>1</td> +<td>1</td> +</tr> +<tr> +<td>3</td> +<td>2</td> +</tr> +<tr> +<td>5</td> +<td>3</td> +</tr> +<tr> +<td>7</td> +<td>4</td> +</tr> +</tbody></table> + +<p><em>Incorrectly setting this flag will cause data corruption to occur!</em></p> + +<h3 id="native_log_file_path"><code>-native_log_file_path</code></h3> + +<p>Location of the Mesos replicated log files. For optimal and consistent performance, consider +allocating a dedicated disk (preferably SSD) for the replicated log. Ensure that this disk is not +used by anything else (e.g. no process logging) and in particular that it is a real disk +and not just a partition.</p> + +<p>Even when a dedicated disk is used, switching from <code>CFQ</code> to <code>deadline</code> I/O scheduler of Linux kernel +can furthermore help with storage performance in Aurora (<a href="https://issues.apache.org/jira/browse/AURORA-1211";>see this ticket for details</a>).</p> + +<h3 id="native_log_zk_group_path"><code>-native_log_zk_group_path</code></h3> + +<p>ZooKeeper path used for Mesos replicated log quorum discovery.</p> + +<p>See <a href="https://github.com/apache/aurora/blob/rel/0.20.0/src/main/java/org/apache/aurora/scheduler/log/mesos/MesosLogStreamModule.java";>code</a> for +other available Mesos replicated log configuration options and default values.</p> + +<h3 id="changing-the-quorum-size">Changing the Quorum Size</h3> + +<p>Special care needs to be taken when changing the size of the Aurora scheduler quorum. +Since Aurora uses a Mesos replicated log, similar steps need to be followed as when +<a href="http://mesos.apache.org/documentation/latest/operational-guide";>changing the Mesos quorum size</a>.</p> + +<p>As a preparation, increase <code>-native_log_quorum_size</code> on each existing scheduler and restart them. +When updating from 3 to 5 schedulers, the quorum size would grow from 2 to 3.</p> + +<p>When starting the new schedulers, use the <code>-native_log_quorum_size</code> set to the new value. Failing to +first increase the quorum size on running schedulers can in some cases result in corruption +or truncating of the replicated log used by Aurora. In that case, see the documentation on +<a href="../backup-restore/">recovering from backup</a>.</p> + +<h2 id="backup-configuration">Backup Configuration</h2> + +<p>Configuration options for the Aurora scheduler backup manager.</p> + +<ul> +<li><code>-backup_interval</code>: The interval on which the scheduler writes local storage backups. +The default is every hour.</li> +<li><code>-backup_dir</code>: Directory to write backups to. As stated above, this should not be co-located on the +same disk as the replicated log.</li> +<li><code>-max_saved_backups</code>: Maximum number of backups to retain before deleting the oldest backup(s).</li> +</ul> + +<h2 id="resource-isolation">Resource Isolation</h2> + +<p>For proper CPU, memory, and disk isolation as mentioned in our <a href="../../features/resource-isolation/">enduser documentation</a>, +we recommend to add the following isolators to the <code>--isolation</code> flag of the Mesos agent:</p> + +<ul> +<li><code>cgroups/cpu</code></li> +<li><code>cgroups/mem</code></li> +<li><code>disk/du</code></li> +</ul> + +<p>In addition, we recommend to set the following <a href="http://mesos.apache.org/documentation/latest/configuration/";>agent flags</a>:</p> + +<ul> +<li><code>--cgroups_limit_swap</code> to enable memory limits on both memory and swap instead of just memory. +Alternatively, you could disable swap on your agent hosts.</li> +<li><code>--cgroups_enable_cfs</code> to enable hard limits on CPU resources via the CFS bandwidth limiting +feature.</li> +<li><code>--enforce_container_disk_quota</code> to enable disk quota enforcement for containers.</li> +</ul> + +<p>To enable the optional GPU support in Mesos, please see the GPU related flags in the +<a href="http://mesos.apache.org/documentation/latest/configuration/";>Mesos configuration</a>. +To enable the corresponding feature in Aurora, you have to start the scheduler with the +flag</p> +<pre class="highlight plaintext"><code>-allow_gpu_resource=true +</code></pre> + +<p>If you want to use revocable resources, first follow the +<a href="http://mesos.apache.org/documentation/latest/oversubscription/";>Mesos oversubscription documentation</a> +and then set set this Aurora scheduler flag to allow receiving revocable Mesos offers:</p> +<pre class="highlight plaintext"><code>-receive_revocable_resources=true +</code></pre> + +<p>Both CPUs and RAM are supported as revocable resources. The former is enabled by the default, +the latter needs to be enabled via:</p> +<pre class="highlight plaintext"><code>-enable_revocable_ram=true +</code></pre> + +<p>Unless you want to use the <a href="https://github.com/apache/aurora/blob/rel/0.20.0/src/main/resources/org/apache/aurora/scheduler/tiers.json";>default</a> +tier configuration, you will also have to specify a file path:</p> +<pre class="highlight plaintext"><code>-tier_config=path/to/tiers/config.json +</code></pre> + +<h2 id="multi-framework-setup">Multi-Framework Setup</h2> + +<p>Aurora holds onto Mesos offers in order to provide efficient scheduling and +<a href="../../features/multitenancy/#preemption">preemption</a>. This is problematic in multi-framework +environments as Aurora might starve other frameworks.</p> + +<p>With a downside of increased scheduling latency, Aurora can be configured to be more cooperative:</p> + +<ul> +<li>Lowering <code>-min_offer_hold_time</code> (e.g. to <code>1mins</code>) can ensure unused offers are returned back to +Mesos more frequently.</li> +<li>Increasing <code>-offer_filter_duration</code> (e.g to <code>30secs</code>) will instruct Mesos +not to re-offer rejected resources for the given duration.</li> +</ul> + +<p>Setting a <a href="http://mesos.apache.org/documentation/latest/quota/";>minimum amount of resources</a> for +each Mesos role can furthermore help to ensure no framework is starved entirely.</p> + +<h2 id="containers">Containers</h2> + +<p>Both the Mesos and Docker containerizers require configuration of the Mesos agent.</p> + +<h3 id="mesos-containerizer">Mesos Containerizer</h3> + +<p>The minimal agent configuration requires to enable Docker and Appc image support for the Mesos +containerizer:</p> +<pre class="highlight plaintext"><code>--containerizers=mesos +--image_providers=appc,docker +--isolation=filesystem/linux,docker/runtime # as an addition to your other isolators +</code></pre> + +<p>Further details can be found in the corresponding <a href="http://mesos.apache.org/documentation/latest/container-image/";>Mesos documentation</a>.</p> + +<h3 id="docker-containerizer">Docker Containerizer</h3> + +<p>The <a href="http://mesos.apache.org/documentation/latest/docker-containerizer/";>Docker containerizer</a> +requires the Docker engine is installed on each agent host. In addition, it must be enabled on the +Mesos agents by launching them with the option:</p> +<pre class="highlight plaintext"><code>--containerizers=mesos,docker +</code></pre> + +<p>If you would like to run a container with a read-only filesystem, it may also be necessary to use +the scheduler flag <code>-thermos_home_in_sandbox</code> in order to set HOME to the sandbox +before the executor runs. This will make sure that the executor/runner PEX extractions happens +inside of the sandbox instead of the container filesystem root.</p> + +<p>If you would like to supply your own parameters to <code>docker run</code> when launching jobs in docker +containers, you may use the following flags:</p> +<pre class="highlight plaintext"><code>-allow_docker_parameters +-default_docker_parameters +</code></pre> + +<p><code>-allow_docker_parameters</code> controls whether or not users may pass their own configuration parameters +through the job configuration files. If set to <code>false</code> (the default), the scheduler will reject +jobs with custom parameters. <em>NOTE</em>: this setting should be used with caution as it allows any job +owner to specify any parameters they wish, including those that may introduce security concerns +(<code>privileged=true</code>, for example).</p> + +<p><code>-default_docker_parameters</code> allows a cluster operator to specify a universal set of parameters that +should be used for every container that does not have parameters explicitly configured at the job +level. The argument accepts a multimap format:</p> +<pre class="highlight plaintext"><code>-default_docker_parameters="read-only=true,tmpfs=/tmp,tmpfs=/run" +</code></pre> + +<h3 id="common-options">Common Options</h3> + +<p>The following Aurora options work for both containerizers.</p> + +<p>A scheduler flag, <code>-global_container_mounts</code> allows mounting paths from the host (i.e the agent machine) +into all containers on that host. The format is a comma separated list of host<em>path:container</em>path[:mode] +tuples. For example <code>-global_container_mounts=/opt/secret_keys_dir:/mnt/secret_keys_dir:ro</code> mounts +<code>/opt/secret_keys_dir</code> from the agents into all launched containers. Valid modes are <code>ro</code> and <code>rw</code>.</p> + +<h2 id="thermos-process-logs">Thermos Process Logs</h2> + +<h3 id="log-destination">Log destination</h3> + +<p>By default, Thermos will write process stdout/stderr to log files in the sandbox. Process object +configuration allows specifying alternate log file destinations like streamed stdout/stderr or +suppression of all log output. Default behavior can be configured for the entire cluster with the +following flag (through the <code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p> +<pre class="highlight plaintext"><code>--runner-logger-destination=both +</code></pre> + +<p><code>both</code> configuration will send logs to files and stream to parent stdout/stderr outputs.</p> + +<p>See <a href="../../reference/configuration/#logger">Configuration Reference</a> for all destination options.</p> + +<h3 id="log-rotation">Log rotation</h3> + +<p>By default, Thermos will not rotate the stdout/stderr logs from child processes and they will grow +without bound. An individual user may change this behavior via configuration on the Process object, +but it may also be desirable to change the default configuration for the entire cluster. +In order to enable rotation by default, the following flags can be applied to Thermos (through the +<code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p> +<pre class="highlight plaintext"><code>--runner-logger-mode=rotate +--runner-rotate-log-size-mb=100 +--runner-rotate-log-backups=10 +</code></pre> + +<p>In the above example, each instance of the Thermos runner will rotate stderr/stdout logs once they +reach 100 MiB in size and keep a maximum of 10 backups. If a user has provided a custom setting for +their process, it will override these default settings.</p> + +<h2 id="thermos-executor-wrapper">Thermos Executor Wrapper</h2> + +<p>If you need to do computation before starting the Thermos executor (for example, setting a different +<code>--announcer-hostname</code> parameter for every executor), then the Thermos executor should be invoked +inside a wrapper script. In such a case, the aurora scheduler should be started with +<code>-thermos_executor_path</code> pointing to the wrapper script and <code>-thermos_executor_resources</code> set to a +comma separated string of all the resources that should be copied into the sandbox (including the +original Thermos executor). Ensure the wrapper script does not access resources outside of the +sandbox, as when the script is run from within a Docker container those resources may not exist.</p> + +<p>For example, to wrap the executor inside a simple wrapper, the scheduler will be started like this +<code>-thermos_executor_path=/path/to/wrapper.sh -thermos_executor_resources=/usr/share/aurora/bin/thermos_executor.pex</code></p> + +<h2 id="custom-executors">Custom Executors</h2> + +<p>The scheduler can be configured to utilize a custom executor by specifying the <code>-custom_executor_config</code> flag. +The flag must be set to the path of a valid executor configuration file.</p> + +<p>For more information on this feature please see the custom executors <a href="../../features/custom-executors/">documentation</a>.</p> + +<h2 id="a-note-on-increasing-executor-overhead">A note on increasing executor overhead</h2> + +<p>Increasing executor overhead on an existing cluster, whether it be for custom executors or for Thermos, +will result in degraded preemption performance until all task which began life with the previous +executor configuration with less overhead are preempted/restarted.</p> + +<h2 id="controlling-mtta-via-update-affinity">Controlling MTTA via Update Affinity</h2> + +<p>When there is high resource contention in your cluster you may experience noticably elevated job update +times, as well as high task churn across the cluster. This is due to Aurora’s first-fit scheduling +algorithm. To alleviate this, you can enable update affinity where the Scheduler will make a best-effort +attempt to reuse the same agent for the updated task (so long as the resources for the job are not being +increased).</p> + +<p>To enable this in the Scheduler, you can set the following options:</p> +<pre class="highlight plaintext"><code>--enable_update_affinity=true +--update_affinity_reservation_hold_time=3mins +</code></pre> + +<p>You will need to tune the hold time to match the behavior you see in your cluster. If you have extremely +high update throughput, you might have to extend it as processing updates could easily add significant +delays between scheduling attempts. You may also have to tune scheduling parameters to achieve the +throughput you need in your cluster. Some relevant settings (with defaults) are:</p> +<pre class="highlight plaintext"><code>--max_schedule_attempts_per_sec=40 +--initial_schedule_penalty=1secs +--max_schedule_penalty=1mins +--scheduling_max_batch_size=3 +--max_tasks_per_schedule_attempt=5 +</code></pre> + +<p>There are metrics exposed by the Scheduler which can provide guidance on where the bottleneck is. +Example metrics to look at:</p> +<pre class="highlight plaintext"><code>- schedule_attempts_blocks (if this number is greater than 0, then task throughput is hitting + limits controlled by --max_scheduler_attempts_per_sec) +- scheduled_task_penalty_* (metrics around scheduling penalties for tasks, if the numbers here are high + then you could have high contention for resources) +</code></pre> + +<p>Most likely you’ll run into limits with the number of update instances that can be processed per minute +before you run into any other limits. So if your total work done per minute starts to exceed 2k instances, +you may need to extend the update<em>affinity</em>reservation<em>hold</em>time.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
