Added: aurora/site/publish/documentation/0.11.0/storage/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.11.0/storage/index.html?rev=1721584&view=auto ============================================================================== --- aurora/site/publish/documentation/0.11.0/storage/index.html (added) +++ aurora/site/publish/documentation/0.11.0/storage/index.html Wed Dec 23 22:45:21 2015 @@ -0,0 +1,202 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/storage/'" + value="0.11.0"> + <option value="0.11.0" + selected="selected"> + 0.11.0 + (latest) + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="aurora-scheduler-storage">Aurora Scheduler Storage</h1> + +<ul> +<li><a href="#overview">Overview</a></li> +<li><a href="#reads-writes-modifications">Reads, writes, modifications</a> + +<ul> +<li><a href="#read-lifecycle">Read lifecycle</a></li> +<li><a href="#write-lifecycle">Write lifecycle</a></li> +</ul></li> +<li><a href="#atomicity-consistency-and-isolation">Atomicity, consistency and isolation</a></li> +<li><a href="#population-on-restart">Population on restart</a></li> +</ul> + +<h2 id="overview">Overview</h2> + +<p>Aurora scheduler maintains data that need to be persisted to survive failovers and restarts. +For example:</p> + +<ul> +<li>Task configurations and scheduled task instances</li> +<li>Job update configurations and update progress</li> +<li>Production resource quotas</li> +<li>Mesos resource offer host attributes</li> +</ul> + +<p>Aurora solves its persistence needs by leveraging the Mesos implementation of a Paxos replicated +log <a href="https://ramcloud.stanford.edu/~ongaro/userstudy/paxos.pdf";>[1]</a> +<a href="http://en.wikipedia.org/wiki/State_machine_replication";>[2]</a> with a key-value +<a href="https://github.com/google/leveldb";>LevelDB</a> storage as persistence media.</p> + +<p>Conceptually, it can be represented by the following major components:</p> + +<ul> +<li>Volatile storage: in-memory cache of all available data. Implemented via in-memory +<a href="http://www.h2database.com/html/main.html";>H2 Database</a> and accessed via +<a href="http://mybatis.github.io/mybatis-3/";>MyBatis</a>.</li> +<li>Log manager: interface between Aurora storage and Mesos replicated log. The default schema format +is <a href="https://github.com/apache/thrift";>thrift</a>. Data is stored in serialized binary form.</li> +<li>Snapshot manager: all data is periodically persisted in Mesos replicated log in a single snapshot. +This helps establishing periodic recovery checkpoints and speeds up volatile storage recovery on +restart.</li> +<li>Backup manager: as a precaution, snapshots are periodically written out into backup files. +This solves a <a href="/documentation/0.11.0/storage-config/#recovering-from-a-scheduler-backup">disaster recovery problem</a> +in case of a complete loss or corruption of Mesos log files.</li> +</ul> + +<p><img alt="Storage hierarchy" src="../images/storage_hierarchy.png" /></p> + +<h2 id="reads-writes-modifications">Reads, writes, modifications</h2> + +<p>All services in Aurora access data via a set of predefined store interfaces (aka stores) logically +grouped by the type of data they serve. Every interface defines a specific set of operations allowed +on the data thus abstracting out the storage access and the actual persistence implementation. The +latter is especially important in view of a general immutability of persisted data. With the Mesos +replicated log as the underlying persistence solution, data can be read and written easily but not +modified. All modifications are simulated by saving new versions of modified objects. This feature +and general performance considerations justify the existence of the volatile in-memory store.</p> + +<h3 id="read-lifecycle">Read lifecycle</h3> + +<p>There are two types of reads available in Aurora: consistent and weakly-consistent. The difference +is explained <a href="#atomicity-and-isolation">below</a>.</p> + +<p>All reads are served from the volatile storage making reads generally cheap storage operations +from the performance standpoint. The majority of the volatile stores are represented by the +in-memory H2 database. This allows for rich schema definitions, queries and relationships that +key-value storage is unable to match.</p> + +<h3 id="write-lifecycle">Write lifecycle</h3> + +<p>Writes are more involved operations since in addition to updating the volatile store data has to be +appended to the replicated log. Data is not available for reads until fully ack-ed by both +replicated log and volatile storage.</p> + +<h2 id="atomicity-consistency-and-isolation">Atomicity, consistency and isolation</h2> + +<p>Aurora uses <a href="http://en.wikipedia.org/wiki/Write-ahead_logging";>write-ahead logging</a> to ensure +consistency between replicated and volatile storage. In Aurora, data is first written into the +replicated log and only then updated in the volatile store.</p> + +<p>Aurora storage uses read-write locks to serialize data mutations and provide consistent view of the +available data. The available <code>Storage</code> interface exposes 3 major types of operations: +* <code>consistentRead</code> - access is locked using reader’s lock and provides consistent view on read +* <code>weaklyConsistentRead</code> - access is lock-less. Delivers best contention performance but may result +in stale reads +* <code>write</code> - access is fully serialized by using writer’s lock. Operation success requires both +volatile and replicated writes to succeed.</p> + +<p>The consistency of the volatile store is enforced via H2 transactional isolation.</p> + +<h2 id="population-on-restart">Population on restart</h2> + +<p>Any time a scheduler restarts, it restores its volatile state from the most recent position recorded +in the replicated log by restoring the snapshot and replaying individual log entries on top to fully +recover the state up to the last write.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
Added: aurora/site/publish/documentation/0.11.0/test-resource-generation/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.11.0/test-resource-generation/index.html?rev=1721584&view=auto ============================================================================== --- aurora/site/publish/documentation/0.11.0/test-resource-generation/index.html (added) +++ aurora/site/publish/documentation/0.11.0/test-resource-generation/index.html Wed Dec 23 22:45:21 2015 @@ -0,0 +1,132 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/test-resource-generation/'" + value="0.11.0"> + <option value="0.11.0" + selected="selected"> + 0.11.0 + (latest) + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="generating-test-resources">Generating test resources</h1> + +<h2 id="background">Background</h2> + +<p>The Aurora source repository and distributions contain several +<a href="https://github.com/apache/aurora/blob/#{git_tag}/src/test/resources/org/apache/thermos/root/checkpoints";>binary files</a>) to +qualify the backwards-compatibility of thermos with checkpoint data. Since +thermos persists state to disk, to be read by the thermos observer), it is important that we have +tests that prevent regressions affecting the ability to parse previously-written data.</p> + +<h2 id="generating-test-files">Generating test files</h2> + +<p>The files included represent persisted checkpoints that exercise different +features of thermos. The existing files should not be modified unless +we are accepting backwards incompatibility, such as with a major release.</p> + +<p>It is not practical to write source code to generate these files on the fly, +as source would be vulnerable to drift (e.g. due to refactoring) in ways +that would undermine the goal of ensuring backwards compatibility.</p> + +<p>The most common reason to add a new checkpoint file would be to provide +coverage for new thermos features that alter the data format. This is +accomplished by writing and running a +<a href="/documentation/0.11.0/configuration-reference/">job configuration</a> that exercises the feature, and +copying the checkpoint file from the sandbox directory, by default this is +<code>/var/run/thermos/checkpoints/<aurora task id></code>.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.11.0/thrift-deprecation/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.11.0/thrift-deprecation/index.html?rev=1721584&view=auto ============================================================================== --- aurora/site/publish/documentation/0.11.0/thrift-deprecation/index.html (added) +++ aurora/site/publish/documentation/0.11.0/thrift-deprecation/index.html Wed Dec 23 22:45:21 2015 @@ -0,0 +1,161 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/thrift-deprecation/'" + value="0.11.0"> + <option value="0.11.0" + selected="selected"> + 0.11.0 + (latest) + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="thrift-api-changes">Thrift API Changes</h1> + +<h2 id="overview">Overview</h2> + +<p>Aurora uses <a href="https://thrift.apache.org/";>Apache Thrift</a> for representing structured data in +client/server RPC protocol as well as for internal data storage. While Thrift is capable of +correctly handling additions and renames of the existing members, field removals must be done +carefully to ensure backwards compatibility and provide predictable deprecation cycle. This +document describes general guidelines for making Thrift schema changes to the existing fields in +<a href="https://github.com/apache/aurora/blob/#{git_tag}/api/src/main/thrift/org/apache/aurora/gen/api.thrift";>api.thrift</a>).</p> + +<p>It is highly recommended to go through the +<a href="http://diwakergupta.github.io/thrift-missing-guide/";>Thrift: The Missing Guide</a> first to refresh on +basic Thrift schema concepts.</p> + +<h2 id="checklist">Checklist</h2> + +<p>Every existing Thrift schema modification is unique in its requirements and must be analyzed +carefully to identify its scope and expected consequences. The following checklist may help in that +analysis: +* Is this a new field/struct? If yes, go ahead +* Is this a pure field/struct rename without any type/structure change? If yes, go ahead and rename +* Anything else, read further to make sure your change is properly planned</p> + +<h2 id="deprecation-cycle">Deprecation cycle</h2> + +<p>Any time a breaking change (e.g.: field replacement or removal) is required, the following cycle +must be followed:</p> + +<h3 id="vcurrent">vCurrent</h3> + +<p>Change is applied in a way that does not break scheduler/client with this version to +communicate with scheduler/client from vCurrent-1. +* Do not remove or rename the old field +* Add a new field as an eventual replacement of the old one and implement a dual read/write +anywhere the old field is used +* Check <a href="https://github.com/apache/aurora/blob/#{git_tag}/api/src/main/thrift/org/apache/aurora/gen/storage.thrift";>storage.thrift</a>) to see if the +affected struct is stored in Aurora scheduler storage. If so, you most likely need to backfill +existing data to ensure both fields are populated eagerly on startup +See <a href="https://github.com/apache/aurora/blob/#{git_tag}/src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java";>StorageBackfill.java</a>) +* Add a deprecation jira ticket into the vCurrent+1 release candidate +* Add a TODO for the deprecated field mentioning the jira ticket</p> + +<h3 id="vcurrent-1">vCurrent+1</h3> + +<p>Finalize the change by removing the deprecated fields from the Thrift schema. +* Drop any dual read/write routines added in the previous version +* Remove the deprecated Thrift field</p> + +<h2 id="testing">Testing</h2> + +<p>It’s always advisable to test your changes in the local vagrant environment to build more +confidence that you change is backwards compatible. It’s easy to simulate different +client/scheduler versions by playing with <code>aurorabuild</code> command. See <a href="/documentation/0.11.0/vagrant/">this document</a> +for more.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.11.0/tutorial/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.11.0/tutorial/index.html?rev=1721584&view=auto ============================================================================== --- aurora/site/publish/documentation/0.11.0/tutorial/index.html (added) +++ aurora/site/publish/documentation/0.11.0/tutorial/index.html Wed Dec 23 22:45:21 2015 @@ -0,0 +1,373 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/tutorial/'" + value="0.11.0"> + <option value="0.11.0" + selected="selected"> + 0.11.0 + (latest) + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h2 id="aurora-tutorial">Aurora Tutorial</h2> + +<ul> +<li><a href="#introduction">Introduction</a></li> +<li><a href="#setup-install-aurora">Setup: Install Aurora</a></li> +<li><a href="#the-script">The Script</a></li> +<li><a href="#aurora-configuration">Aurora Configuration</a></li> +<li><a href="#whats-going-on-in-that-configuration-file">What’s Going On In That Configuration File?</a></li> +<li><a href="#creating-the-job">Creating the Job</a></li> +<li><a href="#watching-the-job-run">Watching the Job Run</a></li> +<li><a href="#cleanup">Cleanup</a></li> +<li><a href="#next-steps">Next Steps</a></li> +</ul> + +<h2 id="introduction">Introduction</h2> + +<p>This tutorial shows how to use the Aurora scheduler to run (and +“<code>printf-debug</code>”) a hello world program on Mesos. The operational +hierarchy is:</p> + +<ul> +<li>Aurora manages and schedules jobs for Mesos to run.</li> +<li>Mesos manages the individual tasks that make up a job.</li> +<li>Thermos manages the individual processes that make up a task.</li> +</ul> + +<p>This is the recommended first Aurora users document to read to start +getting up to speed on the system.</p> + +<p>To get help, email questions to the Aurora Developer List, +<a href="mailto:[email protected]";>[email protected]</a></p> + +<h2 id="setup-install-aurora">Setup: Install Aurora</h2> + +<p>You use the Aurora client and web UI to interact with Aurora jobs. To +install it locally, see <a href="/documentation/0.11.0/vagrant/">vagrant.md</a>. The remainder of this +Tutorial assumes you are running Aurora using Vagrant. Unless otherwise stated, +all commands are to be run from the root of the aurora repository clone.</p> + +<h2 id="the-script">The Script</h2> + +<p>Our “hello world” application is a simple Python script that loops +forever, displaying the time every few seconds. Copy the code below and +put it in a file named <code>hello_world.py</code> in the root of your Aurora repository clone (Note: +this directory is the same as <code>/vagrant</code> inside the Vagrant VMs).</p> + +<p>The script has an intentional bug, which we will explain later on.</p> + +<!-- NOTE: If you are changing this file, be sure to also update examples/vagrant/test_tutorial.sh. +--> +<pre class="highlight python"><code><span style="color: #000000;font-weight: bold">import</span> <span style="color: #555555">sys</span> +<span style="color: #000000;font-weight: bold">import</span> <span style="color: #555555">time</span> + +<span style="color: #000000;font-weight: bold">def</span> <span style="color: #990000;font-weight: bold">main</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">argv</span><span style="background-color: #f8f8f8">):</span> + <span style="background-color: #f8f8f8">SLEEP_DELAY</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">10</span> + <span style="color: #999988;font-style: italic"># Python ninjas - ignore this blatant bug.</span> + <span style="color: #000000;font-weight: bold">for</span> <span style="background-color: #f8f8f8">i</span> <span style="color: #000000;font-weight: bold">in</span> <span style="background-color: #f8f8f8">xrang</span><span style="background-color: #f8f8f8">(</span><span style="color: #009999">100</span><span style="background-color: #f8f8f8">):</span> + <span style="color: #000000;font-weight: bold">print</span><span style="background-color: #f8f8f8">(</span><span style="color: #d14">"Hello world! The time is now: </span><span style="color: #d14">%</span><span style="color: #d14">s. Sleeping for </span><span style="color: #d14">%</span><span style="color: #d14">d secs"</span> <span style="color: #000000;font-weight: bold">%</span> <span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">time</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">asctime</span><span style="background-color: #f8f8f8">(),</span> <span style="background-color: #f8f8f8">SLEEP_DELAY</span><span style="background-color: #f8f8f8">))</span> + <span style="background-color: #f8f8f8">sys</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">stdout</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">flush</span><span style="background-color: #f8f8f8">()</span> + <span style="background-color: #f8f8f8">time</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">sleep</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">SLEEP_DELAY</span><span style="background-color: #f8f8f8">)</span> + +<span style="color: #000000;font-weight: bold">if</span> <span style="background-color: #f8f8f8">__name__</span> <span style="color: #000000;font-weight: bold">==</span> <span style="color: #d14">"__main__"</span><span style="background-color: #f8f8f8">:</span> + <span style="background-color: #f8f8f8">main</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">sys</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">argv</span><span style="background-color: #f8f8f8">)</span> +</code></pre> + +<h2 id="aurora-configuration">Aurora Configuration</h2> + +<p>Once we have our script/program, we need to create a <em>configuration +file</em> that tells Aurora how to manage and launch our Job. Save the below +code in the file <code>hello_world.aurora</code>.</p> + +<!-- NOTE: If you are changing this file, be sure to also update examples/vagrant/test_tutorial.sh. +--> +<pre class="highlight python"><code><span style="background-color: #f8f8f8">pkg_path</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'/vagrant/hello_world.py'</span> + +<span style="color: #999988;font-style: italic"># we use a trick here to make the configuration change with</span> +<span style="color: #999988;font-style: italic"># the contents of the file, for simplicity. in a normal setting, packages would be</span> +<span style="color: #999988;font-style: italic"># versioned, and the version number would be changed in the configuration.</span> +<span style="color: #000000;font-weight: bold">import</span> <span style="color: #555555">hashlib</span> +<span style="color: #000000;font-weight: bold">with</span> <span style="color: #0086B3">open</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">pkg_path</span><span style="background-color: #f8f8f8">,</span> <span style="color: #d14">'rb'</span><span style="background-color: #f8f8f8">)</span> <span style="color: #000000;font-weight: bold">as</span> <span style="background-color: #f8f8f8">f</span><span style="background-color: #f8f8f8">:</span> + <span style="background-color: #f8f8f8">pkg_checksum</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">hashlib</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">md5</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">f</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">read</span><span style="background-color: #f8f8f8">())</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">hexdigest</span><span style="background-color: #f8f8f8">()</span> + +<span style="color: #999988;font-style: italic"># copy hello_world.py into the local sandbox</span> +<span style="background-color: #f8f8f8">install</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">Process</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">name</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'fetch_package'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">cmdline</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'cp </span><span style="color: #d14">%</span><span style="color: #d14">s . && echo </span><span style="color: #d14">%</span><span style="color: #d14">s && chmod +x hello_world.py'</span> <span style="color: #000000;font-weight: bold">%</span> <span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">pkg_path</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">pkg_checksum</span><span style="background-color: #f8f8f8">))</span> + +<span style="color: #999988;font-style: italic"># run the script</span> +<span style="background-color: #f8f8f8">hello_world</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">Process</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">name</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'hello_world'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">cmdline</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'python hello_world.py'</span><span style="background-color: #f8f8f8">)</span> + +<span style="color: #999988;font-style: italic"># describe the task</span> +<span style="background-color: #f8f8f8">hello_world_task</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">SequentialTask</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">processes</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">[</span><span style="background-color: #f8f8f8">install</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">hello_world</span><span style="background-color: #f8f8f8">],</span> + <span style="background-color: #f8f8f8">resources</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">Resources</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">cpu</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">1</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">ram</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">1</span><span style="color: #000000;font-weight: bold">*</span><span style="background-color: #f8f8f8">MB</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">disk</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #009999">8</span><span style="color: #000000;font-weight: bold">*</span><span style="background-color: #f8f8f8">MB</span><span style="background-color: #f8f8f8">) )</span> + +<span style="background-color: #f8f8f8">jobs</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">[</span> + <span style="background-color: #f8f8f8">Service</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">cluster</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'devcluster'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">environment</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'devel'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">role</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'www-data'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">name</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'hello_world'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">task</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">hello_world_task</span><span style="background-color: #f8f8f8">)</span> +<span style="background-color: #f8f8f8">]</span> +</code></pre> + +<p>For more about Aurora configuration files, see the <a href="/documentation/0.11.0/configuration-tutorial/">Configuration +Tutorial</a> and the <a href="/documentation/0.11.0/configuration-reference/">Aurora + Thermos +Reference</a> (preferably after finishing this +tutorial).</p> + +<h2 id="what-39-s-going-on-in-that-configuration-file">What’s Going On In That Configuration File?</h2> + +<p>More than you might think.</p> + +<ol> +<li><p>From a “big picture” viewpoint, it first defines two +Processes. Then it defines a Task that runs the two Processes in the +order specified in the Task definition, as well as specifying what +computational and memory resources are available for them. Finally, +it defines a Job that will schedule the Task on available and suitable +machines. This Job is the sole member of a list of Jobs; you can +specify more than one Job in a config file.</p></li> +<li><p>At the Process level, it specifies how to get your code into the +local sandbox in which it will run. It then specifies how the code is +actually run once the second Process starts.</p></li> +</ol> + +<h2 id="creating-the-job">Creating the Job</h2> + +<p>We’re ready to launch our job! To do so, we use the Aurora Client to +issue a Job creation request to the Aurora scheduler.</p> + +<p>Many Aurora Client commands take a <em>job key</em> argument, which uniquely +identifies a Job. A job key consists of four parts, each separated by a +“/”. The four parts are <code><cluster>/<role>/<environment>/<jobname></code> +in that order. When comparing two job keys, if any of the +four parts is different from its counterpart in the other key, then the +two job keys identify two separate jobs. If all four values are +identical, the job keys identify the same job.</p> + +<p><code>/etc/aurora/clusters.json</code> within the Aurora scheduler has the available +cluster names. For Vagrant, from the top-level of your Aurora repository clone, +do:</p> +<pre class="highlight plaintext"><code>$ vagrant ssh +</code></pre> + +<p>Followed by:</p> +<pre class="highlight plaintext"><code>vagrant@precise64:~$ cat /etc/aurora/clusters.json +</code></pre> + +<p>You’ll see something like:</p> +<pre class="highlight javascript"><code><span style="background-color: #f8f8f8">[{</span> + <span style="color: #d14">"name"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"devcluster"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"zk"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"192.168.33.7"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"scheduler_zk_path"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"/aurora/scheduler"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"auth_mechanism"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"UNAUTHENTICATED"</span> +<span style="background-color: #f8f8f8">}]</span> +</code></pre> + +<p>Use a <code>name</code> value for your job key’s cluster value.</p> + +<p>Role names are user accounts existing on the slave machines. If you don’t know what accounts +are available, contact your sysadmin.</p> + +<p>Environment names are namespaces; you can count on <code>prod</code>, <code>devel</code> and <code>test</code> existing.</p> + +<p>The Aurora Client command that actually runs our Job is <code>aurora job create</code>. It creates a Job as +specified by its job key and configuration file arguments and runs it.</p> +<pre class="highlight plaintext"><code>aurora job create <cluster>/<role>/<environment>/<jobname> <config_file> +</code></pre> + +<p>Or for our example:</p> +<pre class="highlight plaintext"><code>aurora job create devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora +</code></pre> + +<p>This returns:</p> +<pre class="highlight plaintext"><code>$ vagrant ssh +Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64) + + * Documentation: https://help.ubuntu.com/ +Welcome to your Vagrant-built virtual machine. +Last login: Fri Jan 3 02:18:55 2014 from 10.0.2.2 +vagrant@precise64:~$ aurora job create devcluster/www-data/devel/hello_world \ + /vagrant/hello_world.aurora + INFO] Creating job hello_world + INFO] Response from scheduler: OK (message: 1 new tasks pending for job + www-data/devel/hello_world) + INFO] Job url: http://precise64:8081/scheduler/www-data/devel/hello_world +</code></pre> + +<h2 id="watching-the-job-run">Watching the Job Run</h2> + +<p>Now that our job is running, let’s see what it’s doing. Access the +scheduler web interface at <code>http://$scheduler_hostname:$scheduler_port/scheduler</code> +Or when using <code>vagrant</code>, <code>http://192.168.33.7:8081/scheduler</code> +First we see what Jobs are scheduled:</p> + +<p><img alt="Scheduled Jobs" src="../images/ScheduledJobs.png" /></p> + +<p>Click on your user name, which in this case was <code>www-data</code>, and we see the Jobs associated +with that role:</p> + +<p><img alt="Role Jobs" src="../images/RoleJobs.png" /></p> + +<p>If you click on your <code>hello_world</code> Job, you’ll see:</p> + +<p><img alt="hello_world Job" src="../images/HelloWorldJob.png" /></p> + +<p>Oops, looks like our first job didn’t quite work! The task failed, so we have +to figure out what went wrong.</p> + +<p>Access the page for our Task by clicking on its host.</p> + +<p><img alt="Task page" src="../images/TaskBreakdown.png" /></p> + +<p>Once there, we see that the +<code>hello_world</code> process failed. The Task page captures the standard error and +standard output streams and makes them available. Clicking through +to <code>stderr</code> on the failed <code>hello_world</code> process, we see what happened.</p> + +<p><img alt="stderr page" src="../images/stderr.png" /></p> + +<p>It looks like we made a typo in our Python script. We wanted <code>xrange</code>, +not <code>xrang</code>. Edit the <code>hello_world.py</code> script to use the correct function and +we will try again.</p> +<pre class="highlight plaintext"><code>aurora update start devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora +</code></pre> + +<p>This time, the task comes up, we inspect the page, and see that the +<code>hello_world</code> process is running.</p> + +<p><img alt="Running Task page" src="../images/runningtask.png" /></p> + +<p>We then inspect the output by clicking on <code>stdout</code> and see our process’ +output:</p> + +<p><img alt="stdout page" src="../images/stdout.png" /></p> + +<h2 id="cleanup">Cleanup</h2> + +<p>Now that we’re done, we kill the job using the Aurora client:</p> +<pre class="highlight plaintext"><code>vagrant@precise64:~$ aurora job killall devcluster/www-data/devel/hello_world + INFO] Killing tasks for job: devcluster/www-data/devel/hello_world + INFO] Response from scheduler: OK (message: Tasks killed.) + INFO] Job url: http://precise64:8081/scheduler/www-data/devel/hello_world +vagrant@precise64:~$ +</code></pre> + +<p>The job page now shows the <code>hello_world</code> tasks as completed.</p> + +<p><img alt="Killed Task page" src="../images/killedtask.png" /></p> + +<h2 id="next-steps">Next Steps</h2> + +<p>Now that you’ve finished this Tutorial, you should read or do the following:</p> + +<ul> +<li><a href="/documentation/0.11.0/configuration-tutorial/">The Aurora Configuration Tutorial</a>, which provides more examples +and best practices for writing Aurora configurations. You should also look at +the <a href="/documentation/0.11.0/configuration-reference/">Aurora + Thermos Configuration Reference</a>.</li> +<li>The <a href="/documentation/0.11.0/user-guide/">Aurora User Guide</a> provides an overview of how Aurora, Mesos, and +Thermos work “under the hood”.</li> +<li>Explore the Aurora Client - use <code>aurora -h</code>, and read the +<a href="/documentation/0.11.0/client-commands/">Aurora Client Commands</a> document.</li> +</ul> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.11.0/user-guide/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.11.0/user-guide/index.html?rev=1721584&view=auto ============================================================================== --- aurora/site/publish/documentation/0.11.0/user-guide/index.html (added) +++ aurora/site/publish/documentation/0.11.0/user-guide/index.html Wed Dec 23 22:45:21 2015 @@ -0,0 +1,494 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/user-guide/'" + value="0.11.0"> + <option value="0.11.0" + selected="selected"> + 0.11.0 + (latest) + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h2 id="aurora-user-guide">Aurora User Guide</h2> + +<ul> +<li><a href="#overview">Overview</a></li> +<li><a href="#job-lifecycle">Job Lifecycle</a> + +<ul> +<li><a href="#life-of-a-task">Life Of A Task</a></li> +<li><a href="#pending-to-running-states">PENDING to RUNNING states</a></li> +<li><a href="#task-updates">Task Updates</a></li> +<li><a href="#http-health-checking-and-graceful-shutdown">HTTP Health Checking and Graceful Shutdown</a> + +<ul> +<li><a href="#tearing-a-task-down">Tearing a task down</a></li> +</ul></li> +<li><a href="#giving-priority-to-production-tasks-preempting">Giving Priority to Production Tasks: PREEMPTING</a></li> +<li><a href="#natural-termination-finished-failed">Natural Termination: FINISHED, FAILED</a></li> +<li><a href="#forceful-termination-killing-restarting">Forceful Termination: KILLING, RESTARTING</a></li> +</ul></li> +<li><a href="#service-discovery">Service Discovery</a></li> +<li><a href="#configuration">Configuration</a></li> +<li><a href="#creating-jobs">Creating Jobs</a></li> +<li><a href="#interacting-with-jobs">Interacting With Jobs</a></li> +</ul> + +<h2 id="overview">Overview</h2> + +<p>This document gives an overview of how Aurora works under the hood. +It assumes you’ve already worked through the “hello world” example +job in the <a href="/documentation/0.11.0/tutorial/">Aurora Tutorial</a>. Specifics of how to use Aurora are <strong>not</strong> + given here, but pointers to documentation about how to use Aurora are +provided.</p> + +<p>Aurora is a Mesos framework used to schedule <em>jobs</em> onto Mesos. Mesos +cares about individual <em>tasks</em>, but typical jobs consist of dozens or +hundreds of task replicas. Aurora provides a layer on top of Mesos with +its <code>Job</code> abstraction. An Aurora <code>Job</code> consists of a task template and +instructions for creating near-identical replicas of that task (modulo +things like “instance id” or specific port numbers which may differ from +machine to machine).</p> + +<p>How many tasks make up a Job is complicated. On a basic level, a Job consists of +one task template and instructions for creating near-idential replicas of that task +(otherwise referred to as “instances” or “shards”).</p> + +<p>However, since Jobs can be updated on the fly, a single Job identifier or <em>job key</em> +can have multiple job configurations associated with it.</p> + +<p>For example, consider when I have a Job with 4 instances that each +request 1 core of cpu, 1 GB of RAM, and 1 GB of disk space as specified +in the configuration file <code>hello_world.aurora</code>. I want to +update it so it requests 2 GB of RAM instead of 1. I create a new +configuration file to do that called <code>new_hello_world.aurora</code> and +issue a <code>aurora update start <job_key_value>/0-1 new_hello_world.aurora</code> +command.</p> + +<p>This results in instances 0 and 1 having 1 cpu, 2 GB of RAM, and 1 GB of disk space, +while instances 2 and 3 have 1 cpu, 1 GB of RAM, and 1 GB of disk space. If instance 3 +dies and restarts, it restarts with 1 cpu, 1 GB RAM, and 1 GB disk space.</p> + +<p>So that means there are two simultaneous task configurations for the same Job +at the same time, just valid for different ranges of instances.</p> + +<p>This isn’t a recommended pattern, but it is valid and supported by the +Aurora scheduler. This most often manifests in the “canary pattern” where +instance 0 runs with a different configuration than instances 1-N to test +different code versions alongside the actual production job.</p> + +<p>A task can merely be a single <em>process</em> corresponding to a single +command line, such as <code>python2.6 my_script.py</code>. However, a task can also +consist of many separate processes, which all run within a single +sandbox. For example, running multiple cooperating agents together, +such as <code>logrotate</code>, <code>installer</code>, master, or slave processes. This is +where Thermos comes in. While Aurora provides a <code>Job</code> abstraction on +top of Mesos <code>Tasks</code>, Thermos provides a <code>Process</code> abstraction +underneath Mesos <code>Task</code>s and serves as part of the Aurora framework’s +executor.</p> + +<p>You define <code>Job</code>s,<code>Task</code>s, and <code>Process</code>es in a configuration file. +Configuration files are written in Python, and make use of the Pystachio +templating language. They end in a <code>.aurora</code> extension.</p> + +<p>Pystachio is a type-checked dictionary templating library.</p> + +<blockquote> +<p>TL;DR</p> + +<ul> +<li> Aurora manages jobs made of tasks.</li> +<li> Mesos manages tasks made of processes.</li> +<li> Thermos manages processes.</li> +<li> All defined in <code>.aurora</code> configuration file.</li> +</ul> +</blockquote> + +<p><img alt="Aurora hierarchy" src="../images/aurora_hierarchy.png" /></p> + +<p>Each <code>Task</code> has a <em>sandbox</em> created when the <code>Task</code> starts and garbage +collected when it finishes. All of a <code>Task'</code>s processes run in its +sandbox, so processes can share state by using a shared current working +directory.</p> + +<p>The sandbox garbage collection policy considers many factors, most +importantly age and size. It makes a best-effort attempt to keep +sandboxes around as long as possible post-task in order for service +owners to inspect data and logs, should the <code>Task</code> have completed +abnormally. But you can’t design your applications assuming sandboxes +will be around forever, e.g. by building log saving or other +checkpointing mechanisms directly into your application or into your +<code>Job</code> description.</p> + +<h2 id="job-lifecycle">Job Lifecycle</h2> + +<p>When Aurora reads a configuration file and finds a <code>Job</code> definition, it:</p> + +<ol> +<li> Evaluates the <code>Job</code> definition.</li> +<li> Splits the <code>Job</code> into its constituent <code>Task</code>s.</li> +<li> Sends those <code>Task</code>s to the scheduler.</li> +<li> The scheduler puts the <code>Task</code>s into <code>PENDING</code> state, starting each +<code>Task</code>’s life cycle.</li> +</ol> + +<h3 id="life-of-a-task">Life Of A Task</h3> + +<p><img alt="Life of a task" src="../images/lifeofatask.png" /></p> + +<h3 id="pending-to-running-states">PENDING to RUNNING states</h3> + +<p>When a <code>Task</code> is in the <code>PENDING</code> state, the scheduler constantly +searches for machines satisfying that <code>Task</code>’s resource request +requirements (RAM, disk space, CPU time) while maintaining configuration +constraints such as “a <code>Task</code> must run on machines dedicated to a +particular role” or attribute limit constraints such as “at most 2 +<code>Task</code>s from the same <code>Job</code> may run on each rack”. When the scheduler +finds a suitable match, it assigns the <code>Task</code> to a machine and puts the +<code>Task</code> into the <code>ASSIGNED</code> state.</p> + +<p>From the <code>ASSIGNED</code> state, the scheduler sends an RPC to the slave +machine containing <code>Task</code> configuration, which the slave uses to spawn +an executor responsible for the <code>Task</code>’s lifecycle. When the scheduler +receives an acknowledgement that the machine has accepted the <code>Task</code>, +the <code>Task</code> goes into <code>STARTING</code> state.</p> + +<p><code>STARTING</code> state initializes a <code>Task</code> sandbox. When the sandbox is fully +initialized, Thermos begins to invoke <code>Process</code>es. Also, the slave +machine sends an update to the scheduler that the <code>Task</code> is +in <code>RUNNING</code> state.</p> + +<p>If a <code>Task</code> stays in <code>ASSIGNED</code> or <code>STARTING</code> for too long, the +scheduler forces it into <code>LOST</code> state, creating a new <code>Task</code> in its +place that’s sent into <code>PENDING</code> state. This is technically true of any +active state: if the Mesos core tells the scheduler that a slave has +become unhealthy (or outright disappeared), the <code>Task</code>s assigned to that +slave go into <code>LOST</code> state and new <code>Task</code>s are created in their place. +From <code>PENDING</code> state, there is no guarantee a <code>Task</code> will be reassigned +to the same machine unless job constraints explicitly force it there.</p> + +<p>If there is a state mismatch, (e.g. a machine returns from a <code>netsplit</code> +and the scheduler has marked all its <code>Task</code>s <code>LOST</code> and rescheduled +them), a state reconciliation process kills the errant <code>RUNNING</code> tasks, +which may take up to an hour. But to emphasize this point: there is no +uniqueness guarantee for a single instance of a job in the presence of +network partitions. If the Task requires that, it should be baked in at +the application level using a distributed coordination service such as +Zookeeper.</p> + +<h3 id="task-updates">Task Updates</h3> + +<p><code>Job</code> configurations can be updated at any point in their lifecycle. +Usually updates are done incrementally using a process called a <em>rolling +upgrade</em>, in which Tasks are upgraded in small groups, one group at a +time. Updates are done using various Aurora Client commands.</p> + +<p>For a configuration update, the Aurora Client calculates required changes +by examining the current job config state and the new desired job config. +It then starts a rolling batched update process by going through every batch +and performing these operations:</p> + +<ul> +<li>If an instance is present in the scheduler but isn’t in the new config, +then that instance is killed.</li> +<li>If an instance is not present in the scheduler but is present in +the new config, then the instance is created.</li> +<li>If an instance is present in both the scheduler the new config, then +the client diffs both task configs. If it detects any changes, it +performs an instance update by killing the old config instance and adds +the new config instance.</li> +</ul> + +<p>The Aurora client continues through the instance list until all tasks are +updated, in <code>RUNNING,</code> and healthy for a configurable amount of time. +If the client determines the update is not going well (a percentage of health +checks have failed), it cancels the update.</p> + +<p>Update cancellation runs a procedure similar to the described above +update sequence, but in reverse order. New instance configs are swapped +with old instance configs and batch updates proceed backwards +from the point where the update failed. E.g.; (0,1,2) (3,4,5) (6,7, +8-FAIL) results in a rollback in order (8,7,6) (5,4,3) (2,1,0).</p> + +<h3 id="http-health-checking-and-graceful-shutdown">HTTP Health Checking and Graceful Shutdown</h3> + +<p>The Executor implements a protocol for rudimentary control of a task via HTTP. Tasks subscribe for +this protocol by declaring a port named <code>health</code>. Take for example this configuration snippet:</p> +<pre class="highlight plaintext"><code>nginx = Process( + name = 'nginx', + cmdline = './run_nginx.sh -port {{thermos.ports[http]}}') +</code></pre> + +<p>When this Process is included in a job, the job will be allocated a port, and the command line +will be replaced with something like:</p> +<pre class="highlight plaintext"><code>./run_nginx.sh -port 42816 +</code></pre> + +<p>Where 42816 happens to be the allocated. port. Typically, the Executor monitors Processes within +a task only by liveness of the forked process. However, when a <code>health</code> port was allocated, it will +also send periodic HTTP health checks. A task requesting a <code>health</code> port must handle the following +requests:</p> + +<table><thead> +<tr> +<th>HTTP request</th> +<th>Description</th> +</tr> +</thead><tbody> +<tr> +<td><code>GET /health</code></td> +<td>Inquires whether the task is healthy.</td> +</tr> +<tr> +<td><code>POST /quitquitquit</code></td> +<td>Task should initiate graceful shutdown.</td> +</tr> +<tr> +<td><code>POST /abortabortabort</code></td> +<td>Final warning task is being killed.</td> +</tr> +</tbody></table> + +<p>Please see the +<a href="/documentation/0.11.0/configuration-reference/#healthcheckconfig-objects">configuration reference</a> for +configuration options for this feature.</p> + +<h4 id="snoozing-health-checks">Snoozing Health Checks</h4> + +<p>If you need to pause your health check, you can do so by touching a file inside of your sandbox, +named <code>.healthchecksnooze</code></p> + +<p>As long as that file is present, health checks will be disabled, enabling users to gather core dumps +or other performance measurements without worrying about Aurora’s health check killing their +process.</p> + +<p>WARNING: Remember to remove this when you are done, otherwise your instance will have permanently +disabled health checks.</p> + +<h4 id="tearing-a-task-down">Tearing a task down</h4> + +<p>The Executor follows an escalation sequence when killing a running task:</p> + +<ol> +<li>If <code>health</code> port is not present, skip to (5)</li> +<li>POST /quitquitquit</li> +<li>wait 5 seconds</li> +<li>POST /abortabortabort</li> +<li>Send SIGTERM (<code>kill</code>)</li> +<li>Send SIGKILL (<code>kill -9</code>)</li> +</ol> + +<p>If the Executor notices that all Processes in a Task have aborted during this sequence, it will +not proceed with subsequent steps. Note that graceful shutdown is best-effort, and due to the many +inevitable realities of distributed systems, it may not be performed.</p> + +<h3 id="giving-priority-to-production-tasks-preempting">Giving Priority to Production Tasks: PREEMPTING</h3> + +<p>Sometimes a Task needs to be interrupted, such as when a non-production +Task’s resources are needed by a higher priority production Task. This +type of interruption is called a <em>pre-emption</em>. When this happens in +Aurora, the non-production Task is killed and moved into +the <code>PREEMPTING</code> state when both the following are true:</p> + +<ul> +<li>The task being killed is a non-production task.</li> +<li>The other task is a <code>PENDING</code> production task that hasn’t been +scheduled due to a lack of resources.</li> +</ul> + +<p>Since production tasks are much more important, Aurora kills off the +non-production task to free up resources for the production task. The +scheduler UI shows the non-production task was preempted in favor of the +production task. At some point, tasks in <code>PREEMPTING</code> move to <code>KILLED</code>.</p> + +<p>Note that non-production tasks consuming many resources are likely to be +preempted in favor of production tasks.</p> + +<h3 id="natural-termination-finished-failed">Natural Termination: FINISHED, FAILED</h3> + +<p>A <code>RUNNING</code> <code>Task</code> can terminate without direct user interaction. For +example, it may be a finite computation that finishes, even something as +simple as <code>echo hello world.</code>Or it could be an exceptional condition in +a long-lived service. If the <code>Task</code> is successful (its underlying +processes have succeeded with exit status <code>0</code> or finished without +reaching failure limits) it moves into <code>FINISHED</code> state. If it finished +after reaching a set of failure limits, it goes into <code>FAILED</code> state.</p> + +<h3 id="forceful-termination-killing-restarting">Forceful Termination: KILLING, RESTARTING</h3> + +<p>You can terminate a <code>Task</code> by issuing an <code>aurora job kill</code> command, which +moves it into <code>KILLING</code> state. The scheduler then sends the slave a +request to terminate the <code>Task</code>. If the scheduler receives a successful +response, it moves the Task into <code>KILLED</code> state and never restarts it.</p> + +<p>The scheduler has access to a non-public <code>RESTARTING</code> state. If a <code>Task</code> +is forced into the <code>RESTARTING</code> state, the scheduler kills the +underlying task but in parallel schedules an identical replacement for +it.</p> + +<h2 id="configuration">Configuration</h2> + +<p>You define and configure your Jobs (and their Tasks and Processes) in +Aurora configuration files. Their filenames end with the <code>.aurora</code> +suffix, and you write them in Python making use of the Pystachio +templating language, along +with specific Aurora, Mesos, and Thermos commands and methods. See the +<a href="/documentation/0.11.0/configuration-reference/">Configuration Guide and Reference</a> and +<a href="/documentation/0.11.0/configuration-tutorial/">Configuration Tutorial</a>.</p> + +<h2 id="service-discovery">Service Discovery</h2> + +<p>It is possible for the Aurora executor to announce tasks into ServerSets for +the purpose of service discovery. ServerSets use the Zookeeper <a href="http://zookeeper.apache.org/doc/trunk/recipes.html#sc_outOfTheBox";>group membership pattern</a> +of which there are several reference implementations:</p> + +<ul> +<li><a href="https://github.com/apache/mesos/blob/master/src/zookeeper/group.cpp";>C++</a></li> +<li><a href="https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/zookeeper/ServerSetImpl.java#L221";>Java</a></li> +<li><a href="https://github.com/twitter/commons/blob/master/src/python/twitter/common/zookeeper/serverset/serverset.py#L51";>Python</a></li> +</ul> + +<p>These can also be used natively in Finagle using the <a href="https://github.com/twitter/finagle/blob/master/finagle-serversets/src/main/scala/com/twitter/finagle/zookeeper/ZookeeperServerSetCluster.scala";>ZookeeperServerSetCluster</a>.</p> + +<p>For more information about how to configure announcing, see the <a href="/documentation/0.11.0/configuration-reference/">Configuration Reference</a>.</p> + +<h2 id="creating-jobs">Creating Jobs</h2> + +<p>You create and manipulate Aurora Jobs with the Aurora client, which starts all its +command line commands with +<code>aurora</code>. See <a href="/documentation/0.11.0/client-commands/">Aurora Client Commands</a> for details +about the Aurora Client.</p> + +<h2 id="interacting-with-jobs">Interacting With Jobs</h2> + +<p>You interact with Aurora jobs either via:</p> + +<ul> +<li>Read-only Web UIs</li> +</ul> + +<p>Part of the output from creating a new Job is a URL for the Job’s scheduler UI page.</p> + +<p>For example:</p> +<pre class="highlight plaintext"><code> vagrant@precise64:~$ aurora job create devcluster/www-data/prod/hello \ + /vagrant/examples/jobs/hello_world.aurora + INFO] Creating job hello + INFO] Response from scheduler: OK (message: 1 new tasks pending for job www-data/prod/hello) + INFO] Job url: http://precise64:8081/scheduler/www-data/prod/hello +</code></pre> + +<p>The “Job url” goes to the Job’s scheduler UI page. To go to the overall scheduler UI page, + stop at the “scheduler” part of the URL, in this case, <code>http://precise64:8081/scheduler</code></p> + +<p>You can also reach the scheduler UI page via the Client command <code>aurora job open</code>:</p> +<pre class="highlight plaintext"><code> aurora job open [<cluster>[/<role>[/<env>/<job_name>]]] +</code></pre> + +<p>If only the cluster is specified, it goes directly to that cluster’s scheduler main page. + If the role is specified, it goes to the top-level role page. If the full job key is specified, + it goes directly to the job page where you can inspect individual tasks.</p> + +<p>Once you click through to a role page, you see Jobs arranged separately by pending jobs, active + jobs, and finished jobs. Jobs are arranged by role, typically a service account for production + jobs and user accounts for test or development jobs.</p> + +<ul> +<li>The Aurora client</li> +</ul> + +<p>See <a href="/documentation/0.11.0/client-commands/">client commands</a>.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.11.0/vagrant/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.11.0/vagrant/index.html?rev=1721584&view=auto ============================================================================== --- aurora/site/publish/documentation/0.11.0/vagrant/index.html (added) +++ aurora/site/publish/documentation/0.11.0/vagrant/index.html Wed Dec 23 22:45:21 2015 @@ -0,0 +1,232 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/vagrant/'" + value="0.11.0"> + <option value="0.11.0" + selected="selected"> + 0.11.0 + (latest) + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="getting-started">Getting Started</h1> + +<p>This document shows you how to configure a complete cluster using a virtual machine. This setup +replicates a real cluster in your development machine as closely as possible. After you complete +the steps outlined here, you will be ready to create and run your first Aurora job.</p> + +<p>The following sections describe these steps in detail:</p> + +<ol> +<li><a href="#overview">Overview</a></li> +<li><a href="#install-virtualbox-and-vagrant">Install VirtualBox and Vagrant</a></li> +<li><a href="#clone-the-aurora-repository">Clone the Aurora repository</a></li> +<li><a href="#start-the-local-cluster">Start the local cluster</a></li> +<li><a href="#log-onto-the-vm">Log onto the VM</a></li> +<li><a href="#run-your-first-job">Run your first job</a></li> +<li><a href="#rebuild-components">Rebuild components</a></li> +<li><a href="#shut-down-or-delete-your-local-cluster">Shut down or delete your local cluster</a></li> +<li><a href="#troubleshooting">Troubleshooting</a></li> +</ol> + +<h2 id="overview">Overview</h2> + +<p>The Aurora distribution includes a set of scripts that enable you to create a local cluster in +your development machine. These scripts use <a href="https://www.vagrantup.com/";>Vagrant</a> and +<a href="https://www.virtualbox.org/";>VirtualBox</a> to run and configure a virtual machine. Once the +virtual machine is running, the scripts install and initialize Aurora and any required components +to create the local cluster.</p> + +<h2 id="install-virtualbox-and-vagrant">Install VirtualBox and Vagrant</h2> + +<p>First, download and install <a href="https://www.virtualbox.org/";>VirtualBox</a> on your development machine.</p> + +<p>Then download and install <a href="https://www.vagrantup.com/";>Vagrant</a>. To verify that the installation +was successful, open a terminal window and type the <code>vagrant</code> command. You should see a list of +common commands for this tool.</p> + +<h2 id="clone-the-aurora-repository">Clone the Aurora repository</h2> + +<p>To obtain the Aurora source distribution, clone its Git repository using the following command:</p> +<pre class="highlight plaintext"><code> git clone git://git.apache.org/aurora.git +</code></pre> + +<h2 id="start-the-local-cluster">Start the local cluster</h2> + +<p>Now change into the <code>aurora/</code> directory, which contains the Aurora source code and +other scripts and tools:</p> +<pre class="highlight plaintext"><code> cd aurora/ +</code></pre> + +<p>To start the local cluster, type the following command:</p> +<pre class="highlight plaintext"><code> vagrant up +</code></pre> + +<p>This command uses the configuration scripts in the Aurora distribution to:</p> + +<ul> +<li>Download a Linux system image.</li> +<li>Start a virtual machine (VM) and configure it.</li> +<li>Install the required build tools on the VM.</li> +<li>Install Aurora’s requirements (like <a href="http://mesos.apache.org/";>Mesos</a> and +<a href="http://zookeeper.apache.org/";>Zookeeper</a>) on the VM.</li> +<li>Build and install Aurora from source on the VM.</li> +<li>Start Aurora’s services on the VM.</li> +</ul> + +<p>This process takes several minutes to complete.</p> + +<p>To verify that Aurora is running on the cluster, visit the following URLs:</p> + +<ul> +<li>Scheduler - <a href="http://192.168.33.7:8081";>http://192.168.33.7:8081</a></li> +<li>Observer - <a href="http://192.168.33.7:1338";>http://192.168.33.7:1338</a></li> +<li>Mesos Master - <a href="http://192.168.33.7:5050";>http://192.168.33.7:5050</a></li> +<li>Mesos Slave - <a href="http://192.168.33.7:5051";>http://192.168.33.7:5051</a></li> +</ul> + +<h2 id="log-onto-the-vm">Log onto the VM</h2> + +<p>To SSH into the VM, run the following command in your development machine:</p> +<pre class="highlight plaintext"><code> vagrant ssh +</code></pre> + +<p>To verify that Aurora is installed in the VM, type the <code>aurora</code> command. You should see a list +of arguments and possible commands.</p> + +<p>The <code>/vagrant</code> directory on the VM is mapped to the <code>aurora/</code> local directory +from which you started the cluster. You can edit files inside this directory in your development +machine and access them from the VM under <code>/vagrant</code>.</p> + +<p>A pre-installed <code>clusters.json</code> file refers to your local cluster as <code>devcluster</code>, which you +will use in client commands.</p> + +<h2 id="run-your-first-job">Run your first job</h2> + +<p>Now that your cluster is up and running, you are ready to define and run your first job in Aurora. +For more information, see the <a href="/documentation/0.11.0/tutorial/">Aurora Tutorial</a>.</p> + +<h2 id="rebuild-components">Rebuild components</h2> + +<p>If you are changing Aurora code and would like to rebuild a component, you can use the <code>aurorabuild</code> +command on the VM to build and restart a component. This is considerably faster than destroying +and rebuilding your VM.</p> + +<p><code>aurorabuild</code> accepts a list of components to build and update. To get a list of supported +components, invoke the <code>aurorabuild</code> command with no arguments:</p> +<pre class="highlight plaintext"><code> vagrant ssh -c 'aurorabuild client' +</code></pre> + +<h2 id="shut-down-or-delete-your-local-cluster">Shut down or delete your local cluster</h2> + +<p>To shut down your local cluster, run the <code>vagrant halt</code> command in your development machine. To +start it again, run the <code>vagrant up</code> command.</p> + +<p>Once you are finished with your local cluster, or if you would otherwise like to start from scratch, +you can use the command <code>vagrant destroy</code> to turn off and delete the virtual file system.</p> + +<h2 id="troubleshooting">Troubleshooting</h2> + +<p>Most of the vagrant related problems can be fixed by the following steps:</p> + +<ul> +<li>Destroying the vagrant environment with <code>vagrant destroy</code></li> +<li>Killing any orphaned VMs (see AURORA-499) with <code>virtualbox</code> UI or <code>VBoxManage</code> command line tool</li> +<li>Cleaning the repository of build artifacts and other intermediate output with <code>git clean -fdx</code></li> +<li>Bringing up the vagrant environment with <code>vagrant up</code></li> +</ul> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Modified: aurora/site/publish/documentation/0.5.0-incubating/client-commands/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.5.0-incubating/client-commands/index.html?rev=1721584&r1=1721583&r2=1721584&view=diff ============================================================================== --- aurora/site/publish/documentation/0.5.0-incubating/client-commands/index.html (original) +++ aurora/site/publish/documentation/0.5.0-incubating/client-commands/index.html Wed Dec 23 22:45:21 2015 @@ -41,10 +41,14 @@ <h5 class="page-header text-uppercase">Documentation <select onChange="window.location.href='/documentation/' + this.value + '/client-commands/'" value="0.5.0-incubating"> + <option value="0.11.0" + > + 0.11.0 + (latest) + </option> <option value="0.10.0" > 0.10.0 - (latest) </option> <option value="0.9.0" > Modified: aurora/site/publish/documentation/0.5.0-incubating/clientv2/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.5.0-incubating/clientv2/index.html?rev=1721584&r1=1721583&r2=1721584&view=diff ============================================================================== --- aurora/site/publish/documentation/0.5.0-incubating/clientv2/index.html (original) +++ aurora/site/publish/documentation/0.5.0-incubating/clientv2/index.html Wed Dec 23 22:45:21 2015 @@ -41,10 +41,14 @@ <h5 class="page-header text-uppercase">Documentation <select onChange="window.location.href='/documentation/' + this.value + '/clientv2/'" value="0.5.0-incubating"> + <option value="0.11.0" + > + 0.11.0 + (latest) + </option> <option value="0.10.0" > 0.10.0 - (latest) </option> <option value="0.9.0" > Modified: aurora/site/publish/documentation/0.5.0-incubating/committers/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.5.0-incubating/committers/index.html?rev=1721584&r1=1721583&r2=1721584&view=diff ============================================================================== --- aurora/site/publish/documentation/0.5.0-incubating/committers/index.html (original) +++ aurora/site/publish/documentation/0.5.0-incubating/committers/index.html Wed Dec 23 22:45:21 2015 @@ -41,10 +41,14 @@ <h5 class="page-header text-uppercase">Documentation <select onChange="window.location.href='/documentation/' + this.value + '/committers/'" value="0.5.0-incubating"> + <option value="0.11.0" + > + 0.11.0 + (latest) + </option> <option value="0.10.0" > 0.10.0 - (latest) </option> <option value="0.9.0" > Modified: aurora/site/publish/documentation/0.5.0-incubating/configuration-reference/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.5.0-incubating/configuration-reference/index.html?rev=1721584&r1=1721583&r2=1721584&view=diff ============================================================================== --- aurora/site/publish/documentation/0.5.0-incubating/configuration-reference/index.html (original) +++ aurora/site/publish/documentation/0.5.0-incubating/configuration-reference/index.html Wed Dec 23 22:45:21 2015 @@ -41,10 +41,14 @@ <h5 class="page-header text-uppercase">Documentation <select onChange="window.location.href='/documentation/' + this.value + '/configuration-reference/'" value="0.5.0-incubating"> + <option value="0.11.0" + > + 0.11.0 + (latest) + </option> <option value="0.10.0" > 0.10.0 - (latest) </option> <option value="0.9.0" > Modified: aurora/site/publish/documentation/0.5.0-incubating/configuration-tutorial/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.5.0-incubating/configuration-tutorial/index.html?rev=1721584&r1=1721583&r2=1721584&view=diff ============================================================================== --- aurora/site/publish/documentation/0.5.0-incubating/configuration-tutorial/index.html (original) +++ aurora/site/publish/documentation/0.5.0-incubating/configuration-tutorial/index.html Wed Dec 23 22:45:21 2015 @@ -41,10 +41,14 @@ <h5 class="page-header text-uppercase">Documentation <select onChange="window.location.href='/documentation/' + this.value + '/configuration-tutorial/'" value="0.5.0-incubating"> + <option value="0.11.0" + > + 0.11.0 + (latest) + </option> <option value="0.10.0" > 0.10.0 - (latest) </option> <option value="0.9.0" > Modified: aurora/site/publish/documentation/0.5.0-incubating/contributing/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.5.0-incubating/contributing/index.html?rev=1721584&r1=1721583&r2=1721584&view=diff ============================================================================== --- aurora/site/publish/documentation/0.5.0-incubating/contributing/index.html (original) +++ aurora/site/publish/documentation/0.5.0-incubating/contributing/index.html Wed Dec 23 22:45:21 2015 @@ -41,10 +41,14 @@ <h5 class="page-header text-uppercase">Documentation <select onChange="window.location.href='/documentation/' + this.value + '/contributing/'" value="0.5.0-incubating"> + <option value="0.11.0" + > + 0.11.0 + (latest) + </option> <option value="0.10.0" > 0.10.0 - (latest) </option> <option value="0.9.0" > Modified: aurora/site/publish/documentation/0.5.0-incubating/deploying-aurora-scheduler/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.5.0-incubating/deploying-aurora-scheduler/index.html?rev=1721584&r1=1721583&r2=1721584&view=diff ============================================================================== --- aurora/site/publish/documentation/0.5.0-incubating/deploying-aurora-scheduler/index.html (original) +++ aurora/site/publish/documentation/0.5.0-incubating/deploying-aurora-scheduler/index.html Wed Dec 23 22:45:21 2015 @@ -41,10 +41,14 @@ <h5 class="page-header text-uppercase">Documentation <select onChange="window.location.href='/documentation/' + this.value + '/deploying-aurora-scheduler/'" value="0.5.0-incubating"> + <option value="0.11.0" + > + 0.11.0 + (latest) + </option> <option value="0.10.0" > 0.10.0 - (latest) </option> <option value="0.9.0" >
