Modified: aurora/site/publish/documentation/latest/vagrant/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/vagrant/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/vagrant/index.html (original) +++ aurora/site/publish/documentation/latest/vagrant/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -78,17 +78,20 @@ common commands for this tool.</p> <h2 id="clone-the-aurora-repository">Clone the Aurora repository</h2> <p>To obtain the Aurora source distribution, clone its Git repository using the following command:</p> -<pre class="highlight text"> git clone http://git.apache.org/aurora.git -</pre> +<pre class="highlight plaintext"><code> git clone git://git.apache.org/aurora.git +</code></pre> + <h2 id="start-the-local-cluster">Start the local cluster</h2> <p>Now change into the <code>aurora/</code> directory, which contains the Aurora source code and other scripts and tools:</p> -<pre class="highlight text"> cd aurora/ -</pre> +<pre class="highlight plaintext"><code> cd aurora/ +</code></pre> + <p>To start the local cluster, type the following command:</p> -<pre class="highlight text"> vagrant up -</pre> +<pre class="highlight plaintext"><code> vagrant up +</code></pre> + <p>This command uses the configuration scripts in the Aurora distribution to:</p> <ul> @@ -115,8 +118,9 @@ other scripts and tools:</p> <h2 id="log-onto-the-vm">Log onto the VM</h2> <p>To SSH into the VM, run the following command in your development machine:</p> -<pre class="highlight text"> vagrant ssh -</pre> +<pre class="highlight plaintext"><code> vagrant ssh +</code></pre> + <p>To verify that Aurora is installed in the VM, type the <code>aurora</code> command. You should see a list of arguments and possible commands.</p> @@ -140,8 +144,9 @@ and rebuilding your VM.</p> <p><code>aurorabuild</code> accepts a list of components to build and update. To get a list of supported components, invoke the <code>aurorabuild</code> command with no arguments:</p> -<pre class="highlight text"> vagrant ssh -c 'aurorabuild client' -</pre> +<pre class="highlight plaintext"><code> vagrant ssh -c 'aurorabuild client' +</code></pre> + <h2 id="shut-down-or-delete-your-local-cluster">Shut down or delete your local cluster</h2> <p>To shut down your local cluster, run the <code>vagrant halt</code> command in your development machine. To @@ -160,11 +165,11 @@ you can use the command <code>vagrant de <li>Cleaning the repository of build artifacts and other intermediate output with <code>git clean -fdx</code></li> <li>Bringing up the vagrant environment with <code>vagrant up</code></li> </ul> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -189,5 +194,6 @@ you can use the command <code>vagrant de </div> </div> </div> + </body> </html> \ No newline at end of file
Modified: aurora/site/publish/downloads/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/downloads/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/downloads/index.html (original) +++ aurora/site/publish/downloads/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <h1 id="download-aurora">Download Aurora</h1> @@ -84,15 +84,15 @@ <h2 id="git">Git</h2> <p>The latest code is available in git via:</p> -<pre class="highlight text">git clone git://git.apache.org/aurora.git -</pre> +<pre class="highlight plaintext"><code>git clone git://git.apache.org/aurora.git +</code></pre> + <p>You can browse the repo on <a href="https://git-wip-us.apache.org/repos/asf?p=aurora.git";>Apache Git</a> and the <a href="https://github.com/apache/aurora";>GitHub mirror</a>.</p> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -117,5 +117,6 @@ and the <a href="https://github.com/apac </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/index.html (original) +++ aurora/site/publish/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-homepage-header"> <div class="container"> <div class="nav nav-bar"> <img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -41,9 +40,11 @@ <h1>Apache Aurora is a Mesos framework for long-running services and cron jobs.</h1> <br /><br /><br /><br /><br /><br /><br /> </div> + +<p></div> </div> -</div> -</div> +</div></p> + <div class="container-fluid section-ltgreen buffer"> <div class="container"> <div class="row"> @@ -75,7 +76,6 @@ </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -100,5 +100,6 @@ </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/sitemap.xml URL: http://svn.apache.org/viewvc/aurora/site/publish/sitemap.xml?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/sitemap.xml (original) +++ aurora/site/publish/sitemap.xml Sat Dec 12 01:46:48 2015 @@ -2,158 +2,158 @@ <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";> <url> <loc>http://aurora.apache.org/blog/aurora-0-6-0-incubating-released/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/blog/aurora-0-7-0-incubating-released/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/blog/2015-upcoming-apache-aurora-meetups/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/blog/aurora-0-8-0-released/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/blog/aurora-0-9-0-released/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/blog/aurora-at-mesoscon-seattle/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/blog/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/community/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/developers/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/docs/gettingstarted/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/docs/howtocontribute/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/build-system/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/client-cluster-configuration/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/client-commands/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/committers/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/configuration-reference/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/configuration-tutorial/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/contributing/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/cron-jobs/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/deploying-aurora-scheduler/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/developing-aurora-client/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/developing-aurora-scheduler/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/hooks/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/monitoring/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/presentations/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/resources/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/scheduler-storage/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/security/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/sla/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/storage-config/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/storage/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/test-resource-generation/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/thrift-deprecation/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/tutorial/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/user-guide/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/vagrant/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/documentation/latest/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/downloads/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> <url> <loc>http://aurora.apache.org/</loc> - <lastmod>2015-11-17T00:00:00-08:00</lastmod> + <lastmod>2015-12-12T00:00:00+00:00</lastmod> </url> </urlset> \ No newline at end of file Copied: aurora/site/source/_footer.erb (from r1719615, aurora/site/source/_footer.md.erb) URL: http://svn.apache.org/viewvc/aurora/site/source/_footer.erb?p2=aurora/site/source/_footer.erb&p1=aurora/site/source/_footer.md.erb&r1=1719615&r2=1719617&rev=1719617&view=diff ============================================================================== --- aurora/site/source/_footer.md.erb (original) +++ aurora/site/source/_footer.erb Sat Dec 12 01:46:48 2015 @@ -21,4 +21,4 @@ <p class="disclaimer">Copyright 2014 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> </div> </div> - </div> \ No newline at end of file + </div> Modified: aurora/site/source/documentation/latest.html.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest.html.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest.html.md (original) +++ aurora/site/source/documentation/latest.html.md Sat Dec 12 01:46:48 2015 @@ -23,11 +23,15 @@ We encourage you to ask questions on the * [Scheduler Storage](/documentation/latest/storage/) * [Scheduler Storage and Maintenance](/documentation/latest/storage-config/) * [SLA Measurement](/documentation/latest/sla/) - * [Resource Isolation and Sizing](/documentation/latest/resource-isolation/) + * [Resource Isolation and Sizing](/documentation/latest/resources/) * [Generating test resources](/documentation/latest/test-resource-generation/) ## Developers - * [Contributing to the project](/documentation/latest/contributing/) + * [Contributing to the project](../CONTRIBUTING.md) * [Developing the Aurora Scheduler](/documentation/latest/developing-aurora-scheduler/) * [Developing the Aurora Client](/documentation/latest/developing-aurora-client/) * [Committers Guide](/documentation/latest/committers/) + * [Build System](/documentation/latest/build-system/) + +## Additional Resources + * [Presentation videos and slides](/documentation/latest/presentations/) Modified: aurora/site/source/documentation/latest/client-commands.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/client-commands.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/client-commands.md (original) +++ aurora/site/source/documentation/latest/client-commands.md Sat Dec 12 01:46:48 2015 @@ -332,7 +332,8 @@ configuration file, and displays the par aurora quota get CLUSTER/ROLE Prints the production quota allocated to the role's value at the given -cluster. +cluster. Only non-[dedicated](deploying-aurora-scheduler.md#dedicated-attribute) +[production](configuration-reference.md#job-objects) jobs consume quota. ### Finding a Job on Web UI Modified: aurora/site/source/documentation/latest/configuration-reference.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/configuration-reference.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/configuration-reference.md (original) +++ aurora/site/source/documentation/latest/configuration-reference.md Sat Dec 12 01:46:48 2015 @@ -26,10 +26,12 @@ Aurora + Thermos Configuration Reference - [Job Schema](#job-schema) - [Job Objects](#job-objects) - [Services](#services) + - [Revocable Jobs](#revocable-jobs) - [UpdateConfig Objects](#updateconfig-objects) - [HealthCheckConfig Objects](#healthcheckconfig-objects) - [Announcer Objects](#announcer-objects) - [Container Objects](#container) + - [LifecycleConfig Objects](#lifecycleconfig-objects) - [Specifying Scheduling Constraints](#specifying-scheduling-constraints) - [Template Namespaces](#template-namespaces) - [mesos Namespace](#mesos-namespace) @@ -196,6 +198,11 @@ is shorthand for [Constraint(order = [process1.name(), process2.name()])] +The `order` function accepts Process name strings `('foo', 'bar')` or the processes +themselves, e.g. `foo=Process(name='foo', ...)`, `bar=Process(name='bar', ...)`, +`constraints=order(foo, bar)`. + + #### resources Takes a `Resource` object, which specifies the amounts of CPU, memory, and disk space resources @@ -203,10 +210,10 @@ to allocate to the Task. #### max_failures -`max_failures` is the number of times processes that are part of this -Task can fail before the entire Task is marked for failure. +`max_failures` is the number of failed processes needed for the `Task` to be +marked as failed. -For example: +For example, assume a Task has two Processes and a `max_failures` value of `2`: template = Process(max_failures=10) task = Task( @@ -217,11 +224,13 @@ For example: ], max_failures=2) -The `failing` Process could fail 10 times before being marked as -permanently failed, and the `succeeding` Process would succeed on the -first run. The task would succeed despite only allowing for two failed -processes. To be more specific, there would be 10 failed process runs -yet 1 failed process. +The `failing` Process could fail 10 times before being marked as permanently +failed, and the `succeeding` Process could succeed on the first run. However, +the task would succeed despite only allowing for two failed processes. To be more +specific, there would be 10 failed process runs yet 1 failed process. Both processes +would have to fail for the Task to fail. + + #### max_concurrency @@ -277,6 +286,7 @@ Client applications with higher priority finalization wait (e.g. through parameters to `thermos kill`), so this is mostly a best-effort signal. + ### Constraint Object Current constraint objects only support a single ordering constraint, `order`, @@ -291,7 +301,7 @@ ordering constraints. ### Resource Object Specifies the amount of CPU, Ram, and disk resources the task needs. See the -[Resource Isolation document](/documentation/latest/resource-isolation/) for suggested values and to understand how +[Resource Isolation document](/documentation/latest/resources/) for suggested values and to understand how resources are allocated. param | type | description @@ -317,14 +327,16 @@ Job Schema ```instances```| Integer | Number of instances (sometimes referred to as replicas or shards) of the task to create. (Default: 1) ```cron_schedule``` | String | Cron schedule in cron format. May only be used with non-service jobs. See [Cron Jobs](/documentation/latest/cron-jobs/) for more information. Default: None (not a cron job.) ```cron_collision_policy``` | String | Policy to use when a cron job is triggered while a previous run is still active. KILL_EXISTING Kill the previous run, and schedule the new run CANCEL_NEW Let the previous run continue, and cancel the new run. (Default: KILL_EXISTING) - ```update_config``` | ```update_config``` object | Parameters for controlling the rate and policy of rolling updates. + ```update_config``` | ```UpdateConfig``` object | Parameters for controlling the rate and policy of rolling updates. ```constraints``` | dict | Scheduling constraints for the tasks. See the section on the [constraint specification language](#Specifying-Scheduling-Constraints) ```service``` | Boolean | If True, restart tasks regardless of success or failure. (Default: False) ```max_task_failures``` | Integer | Maximum number of failures after which the task is considered to have failed (Default: 1) Set to -1 to allow for infinite failures ```priority``` | Integer | Preemption priority to give the task (Default 0). Tasks with higher priorities may preempt tasks at lower priorities. - ```production``` | Boolean | Whether or not this is a production task backed by quota (Default: False). Production jobs may preempt any non-production job, and may only be preempted by production jobs in the same role and of higher priority. To run jobs at this level, the job role must have the appropriate quota. To grant quota to a particular role in production, operators use the ``aurora_admin set_quota`` command. - ```health_check_config``` | ```heath_check_config``` object | Parameters for controlling a task's health checks via HTTP. Only used if a health port was assigned with a command line wildcard. + ```production``` | Boolean | Whether or not this is a production task that may [preempt](resources.md#task-preemption) other tasks (Default: False). Production job role must have the appropriate [quota](resources.md#resource-quota). + ```health_check_config``` | ```HealthCheckConfig``` object | Parameters for controlling a task's health checks via HTTP. Only used if a health port was assigned with a command line wildcard. ```container``` | ```Container``` object | An optional container to run all processes inside of. + ```lifecycle``` | ```LifecycleConfig``` object | An optional task lifecycle configuration that dictates commands to be executed on startup/teardown. HTTP lifecycle is enabled by default if the "health" port is requested. See [LifecycleConfig Objects](#lifecycleconfig-objects) for more information. + ```tier``` | String | Task tier type. When set to `revocable` requires the task to run with Mesos revocable resources. This is work [in progress](https://issues.apache.org/jira/browse/AURORA-1343) and is currently only supported for the revocable tasks. The ultimate goal is to simplify task configuration by hiding various configuration knobs behind a task tier definition. See AURORA-1343 and AURORA-1443 for more details. ### Services @@ -336,6 +348,21 @@ Jobs without the service bit set only re `max_task_failures` times and only if they terminated unsuccessfully either due to human error or machine failure. +### Revocable Jobs + +**WARNING**: This feature is currently in alpha status. Do not use it in production clusters! + +Mesos [supports a concept of revocable tasks](http://mesos.apache.org/documentation/latest/oversubscription/) +by oversubscribing machine resources by the amount deemed safe to not affect the existing +non-revocable tasks. Aurora now supports revocable jobs via a `tier` setting set to `revocable` +value. + +More implementation details in this [ticket](https://issues.apache.org/jira/browse/AURORA-1343). + +Scheduler must be [configured](deploying-aurora-scheduler.md#configuring-resource-oversubscription) +to receive revocable offers from Mesos and accept revocable jobs. If not configured properly +revocable tasks will never get assigned to hosts and will stay in PENDING. + ### UpdateConfig Objects Parameters for controlling the rate and policy of rolling updates. @@ -359,8 +386,11 @@ Parameters for controlling a task's heal | ------- | :-------: | -------- | ```initial_interval_secs``` | Integer | Initial delay for performing an HTTP health check. (Default: 15) | ```interval_secs``` | Integer | Interval on which to check the task's health via HTTP. (Default: 10) -| ```timeout_secs``` | Integer | HTTP request timeout. (Default: 1) | ```max_consecutive_failures``` | Integer | Maximum number of consecutive failures that tolerated before considering a task unhealthy (Default: 0) +| ```timeout_secs``` | Integer | HTTP request timeout. (Default: 1) +| ```endpoint``` | String | HTTP endpoint to check (Default: /health) +| ```expected_response``` | String | If not empty, fail the health check if the response differs. Case insensitive. (Default: ok) +| ```expected_response_code``` | Integer | If not zero, fail the health check if the response code differs. (Default: 0) ### Announcer Objects @@ -411,9 +441,51 @@ Describes the container the job's proces ### Docker Object - param | type | description - ----- | :----: | ----------- - ```image``` | String | The name of the docker image to execute. If the image does not exist locally it will be pulled with ```docker pull```. + param | type | description + ----- | :----: | ----------- + ```image``` | String | The name of the docker image to execute. If the image does not exist locally it will be pulled with ```docker pull```. + ```parameters``` | List(Parameter) | Additional parameters to pass to the docker containerizer. + +### Docker Parameter Object + +Docker CLI parameters. This needs to be enabled by the scheduler `enable_docker_parameters` option. +See [Docker Command Line Reference](https://docs.docker.com/reference/commandline/run/) for valid parameters. + + param | type | description + ----- | :----: | ----------- + ```name``` | String | The name of the docker parameter. E.g. volume + ```value``` | String | The value of the parameter. E.g. /usr/local/bin:/usr/bin:rw + +### LifecycleConfig Objects + +*Note: The only lifecycle configuration supported is the HTTP lifecycle via the HTTPLifecycleConfig.* + + param | type | description + ----- | :----: | ----------- + ```http``` | HTTPLifecycleConfig | Configure the lifecycle manager to send lifecycle commands to the task via HTTP. + +### HTTPLifecycleConfig Objects + + param | type | description + ----- | :----: | ----------- + ```port``` | String | The named port to send POST commands (Default: health) + ```graceful_shutdown_endpoint``` | String | Endpoint to hit to indicate that a task should gracefully shutdown. (Default: /quitquitquit) + ```shutdown_endpoint``` | String | Endpoint to hit to give a task its final warning before being killed. (Default: /abortabortabort) + +#### graceful_shutdown_endpoint + +If the Job is listening on the port as specified by the HTTPLifecycleConfig +(default: `health`), a HTTP POST request will be sent over localhost to this +endpoint to request that the task gracefully shut itself down. This is a +courtesy call before the `shutdown_endpoint` is invoked a fixed amount of +time later. + +#### shutdown_endpoint + +If the Job is listening on the port as specified by the HTTPLifecycleConfig +(default: `health`), a HTTP POST request will be sent over localhost to this +endpoint to request as a final warning before being shut down. If the task +does not shut down on its own after this, it will be forcefully killed Specifying Scheduling Constraints Modified: aurora/site/source/documentation/latest/configuration-tutorial.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/configuration-tutorial.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/configuration-tutorial.md (original) +++ aurora/site/source/documentation/latest/configuration-tutorial.md Sat Dec 12 01:46:48 2015 @@ -236,57 +236,8 @@ above multiple `Process` definitions int run_task = SequentialTask(processes = [stage, run]) -`Process` also has five optional attributes, each with a default value -if one isn't specified in the configuration: +`Process` also has optional attributes to customize its behaviour. Details can be found in the [*Aurora+Thermos Configuration Reference*](configuration-reference.md#process-objects). -- `max_failures`: Defaulting to `1`, the maximum number of failures - (non-zero exit statuses) before this `Process` is marked permanently - failed and not retried. If a `Process` permanently fails, Thermos - checks the `Process` object's containing `Task` for the task's - failure limit (usually 1) to determine whether or not the `Task` - should be failed. Setting `max_failures`to `0` means that this - process will keep retrying until a successful (zero) exit status is - achieved. Retries happen at most once every `min_duration` seconds - to prevent effectively mounting a denial of service attack against - the coordinating scheduler. - -- `daemon`: Defaulting to `False`, if `daemon` is set to `True`, a - successful (zero) exit status does not prevent future process runs. - Instead, the `Process` reinvokes after `min_duration` seconds. - However, the maximum failure limit (`max_failures`) still - applies. A combination of `daemon=True` and `max_failures=0` retries - a `Process` indefinitely regardless of exit status. This should - generally be avoided for very short-lived processes because of the - accumulation of checkpointed state for each process run. When - running in Aurora, `max_failures` is capped at - 100. - -- `ephemeral`: Defaulting to `False`, if `ephemeral` is `True`, the - `Process`' status is not used to determine if its bound `Task` has - completed. For example, consider a `Task` with a - non-ephemeral webserver process and an ephemeral logsaver process - that periodically checkpoints its log files to a centralized data - store. The `Task` is considered finished once the webserver process - finishes, regardless of the logsaver's current status. - -- `min_duration`: Defaults to `15`. Processes may succeed or fail - multiple times during a single Task. Each result is called a - *process run* and this value is the minimum number of seconds the - scheduler waits before re-running the same process. - -- `final`: Defaulting to `False`, this is a finalizing `Process` that - should run last. Processes can be grouped into two classes: - *ordinary* and *finalizing*. By default, Thermos Processes are - ordinary. They run as long as the `Task` is considered - healthy (i.e. hasn't reached a failure limit). But once all regular - Thermos Processes have either finished or the `Task` has reached a - certain failure threshold, Thermos moves into a *finalization* stage - and runs all finalizing Processes. These are typically necessary for - cleaning up after the `Task`, such as log checkpointers, or perhaps - e-mail notifications of a completed Task. Finalizing processes may - not depend upon ordinary processes or vice-versa, however finalizing - processes may depend upon other finalizing processes and will - otherwise run as a typical process schedule. ## Getting Your Code Into The Sandbox @@ -346,59 +297,8 @@ A basic Task definition looks like: ram = 1*GB, disk = 1*GB)) -There are four optional Task attributes: +A Task has optional attributes to customize its behaviour. Details can be found in the [*Aurora+Thermos Configuration Reference*](configuration-reference.md#task-object) -- `constraints`: A list of `Constraint` objects that constrain the - Task's processes. Currently there is only one type, the `order` - constraint. For example the following requires that the processes - run in the order `foo`, then `bar`. - - constraints = [Constraint(order=['foo', 'bar'])] - - There is an `order()` function that takes `order('foo', 'bar', 'baz')` - and converts it into `[Constraint(order=['foo', 'bar', 'baz'])]`. - `order()` accepts Process name strings `('foo', 'bar')` or the processes - themselves, e.g. `foo=Process(name='foo', ...)`, `bar=Process(name='bar', ...)`, - `constraints=order(foo, bar)` - - Note that Thermos rejects tasks with process cycles. - -- `max_failures`: Defaulting to `1`, the number of failed processes - needed for the `Task` to be marked as failed. Note how this - interacts with individual Processes' `max_failures` values. Assume a - Task has two Processes and a `max_failures` value of `2`. So both - Processes must fail for the Task to fail. Now, assume each of the - Task's Processes has its own `max_failures` value of `10`. If - Process "A" fails 5 times before succeeding, and Process "B" fails - 10 times and is then marked as failing, their parent Task succeeds. - Even though there were 15 individual failures by its Processes, only - 1 of its Processes was finally marked as failing. Since 1 is less - than the 2 that is the Task's `max_failures` value, the Task does - not fail. - -- `max_concurrency`: Defaulting to `0`, the maximum number of - concurrent processes in the Task. `0` specifies unlimited - concurrency. For Tasks with many expensive but otherwise independent - processes, you can limit the amount of concurrency Thermos schedules - instead of artificially constraining them through `order` - constraints. For example, a test framework may generate a Task with - 100 test run processes, but runs it in a Task with - `resources.cpus=4`. Limit the amount of parallelism to 4 by setting - `max_concurrency=4`. - -- `finalization_wait`: Defaulting to `30`, the number of seconds - allocated for finalizing the Task's processes. A Task starts in - `ACTIVE` state when Processes run and stays there as long as the Task - is healthy and Processes run. When all Processes finish successfully - or the Task reaches its maximum process failure limit, it goes into - `CLEANING` state. In `CLEANING`, it sends `SIGTERMS` to any still running - Processes. When all Processes terminate, the Task goes into - `FINALIZING` state and invokes the schedule of all processes whose - final attribute has a True value. Everything from the end of `ACTIVE` - to the end of `FINALIZING` must happen within `finalization_wait` - number of seconds. If not, all still running Processes are sent - `SIGKILL`s (or if dependent on yet to be completed Processes, are - never invoked). ### SequentialTask: Running Processes in Parallel or Sequentially @@ -554,82 +454,8 @@ default. For these four parameters, a Jo task = foo_task) In addition to the required attributes, there are several optional -attributes. The first (strongly recommended) optional attribute is: +attributes. Details can be found in the [Aurora+Thermos Configuration Reference](configuration-reference.md#job-objects). -- `contact`: An email address for the Job's owner. For production - jobs, it is usually a team mailing list. - -Two more attributes deal with how to handle failure of the Job's Task: - -- `max_task_failures`: An integer, defaulting to `1`, of the maximum - number of Task failures after which the Job is considered failed. - `-1` allows for infinite failures. - -- `service`: A boolean, defaulting to `False`, which if `True` - restarts tasks regardless of whether they succeeded or failed. In - other words, if `True`, after the Job's Task completes, it - automatically starts again. This is for Jobs you want to run - continuously, rather than doing a single run. - -Three attributes deal with configuring the Job's Task: - -- `instances`: Defaulting to `1`, the number of - instances/replicas/shards of the Job's Task to create. - -- `priority`: Defaulting to `0`, the Job's Task's preemption priority, - for which higher values may preempt Tasks from Jobs with lower - values. - -- `production`: a Boolean, defaulting to `False`, specifying that this - is a production job backed by quota. Tasks from production Jobs may - preempt tasks from any non-production job, and may only be preempted - by tasks from production jobs in the same role with higher - priority. **WARNING**: To run Jobs at this level, the Job role must - have the appropriate quota. To grant quota to a particular role in - production, operators use the ``aurora_admin set_quota`` command. - -The final three Job attributes each take an object as their value. - -- `update_config`: An `UpdateConfig` - object provides parameters for controlling the rate and policy of - rolling updates. The `UpdateConfig` parameters are: - - `batch_size`: An integer, defaulting to `1`, specifying the - maximum number of shards to update in one iteration. - - `restart_threshold`: An integer, defaulting to `60`, specifying - the maximum number of seconds before a shard must move into the - `RUNNING` state before considered a failure. - - `watch_secs`: An integer, defaulting to `45`, specifying the - minimum number of seconds a shard must remain in the `RUNNING` - state before considered a success. - - `max_per_shard_failures`: An integer, defaulting to `0`, - specifying the maximum number of restarts per shard during an - update. When the limit is exceeded, it increments the total - failure count. - - `max_total_failures`: An integer, defaulting to `0`, specifying - the maximum number of shard failures tolerated during an update. - Cannot be equal to or greater than the job's total number of - tasks. -- `health_check_config`: A `HealthCheckConfig` object that provides - parameters for controlling a Task's health checks via HTTP. Only - used if a health port was assigned with a command line wildcard. The - `HealthCheckConfig` parameters are: - - `initial_interval_secs`: An integer, defaulting to `15`, - specifying the initial delay for doing an HTTP health check. - - `interval_secs`: An integer, defaulting to `10`, specifying the - number of seconds in the interval between checking the Task's - health. - - `timeout_secs`: An integer, defaulting to `1`, specifying the - number of seconds the application must respond to an HTTP health - check with `OK` before it is considered a failure. - - `max_consecutive_failures`: An integer, defaulting to `0`, - specifying the maximum number of consecutive failures before a - task is unhealthy. -- `constraints`: A `dict` Python object, specifying Task scheduling - constraints. Most users will not need to specify constraints, as the - scheduler automatically inserts reasonable defaults. Please do not - set this field unless you are sure of what you are doing. See the - section in the Aurora + Thermos Reference manual on [Specifying - Scheduling Constraints](/documentation/latest/configuration-reference/) for more information. ## The jobs List Modified: aurora/site/source/documentation/latest/contributing.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/contributing.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/contributing.md (original) +++ aurora/site/source/documentation/latest/contributing.md Sat Dec 12 01:46:48 2015 @@ -34,6 +34,9 @@ Post a review with `rbt`, fill out the f ./rbt post -o +If you're unsure about who to add as a reviewer, you can default to adding Bill Farner (wfarner) and +Joshua Cohen (jcohen). They will take care of finding an appropriate reviewer for the patch. + Once you've done this, you probably want to mark the associated Jira issue as Reviewable. Updating an Existing Review Modified: aurora/site/source/documentation/latest/cron-jobs.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/cron-jobs.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/cron-jobs.md (original) +++ aurora/site/source/documentation/latest/cron-jobs.md Sat Dec 12 01:46:48 2015 @@ -67,7 +67,7 @@ grow faster than they can process it. Unlike with services, which aurora will always re-execute regardless of exit status, instances of cron jobs retry according to the `max_task_failures` attribute of the -[Task](configuration-reference.md#task-objects) object. To get "run-until-failure" semantics, +[Task](configuration-reference.md#task-objects) object. To get "run-until-success" semantics, set `max_task_failures` to `-1`. ## Interacting with cron jobs via the Aurora CLI Modified: aurora/site/source/documentation/latest/deploying-aurora-scheduler.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/deploying-aurora-scheduler.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/deploying-aurora-scheduler.md (original) +++ aurora/site/source/documentation/latest/deploying-aurora-scheduler.md Sat Dec 12 01:46:48 2015 @@ -13,6 +13,8 @@ machines. This guide helps you get the - [Storage Performance Considerations](#storage-performance-considerations) - [Network considerations](#network-considerations) - [Considerations for running jobs in docker](#considerations-for-running-jobs-in-docker) + - [Security Considerations](#security-considerations) + - [Configuring Resource Oversubscription](#configuring-resource-oversubscription) - [Running Aurora](#running-aurora) - [Maintaining an Aurora Installation](#maintaining-an-aurora-installation) - [Monitoring](#monitoring) @@ -20,6 +22,8 @@ machines. This guide helps you get the - [Dedicated attribute](#dedicated-attribute) - [Syntax](#syntax) - [Example](#example) +- [Best practices](#best-practices) + - [Diversity](#diversity) - [Common problems](#common-problems) - [Replicated log not initialized](#replicated-log-not-initialized) - [Symptoms](#symptoms) @@ -27,14 +31,14 @@ machines. This guide helps you get the - [Scheduler not registered](#scheduler-not-registered) - [Symptoms](#symptoms-1) - [Solution](#solution-1) - - [Tasks are stuck in PENDING forever](#tasks-are-stuck-in-pending-forever) - - [Symptoms](#symptoms-2) - - [Solution](#solution-2) +- [Changing Scheduler Quorum Size](#changing-scheduler-quorum-size) + - [Preparation](#preparation) + - [Adding New Schedulers](#adding-new-schedulers) ## Installing Aurora The Aurora scheduler is a standalone Java server. As part of the build process it creates a bundle of all its dependencies, with the notable exceptions of the JVM and libmesos. Each target server -should have a JVM (Java 7 or higher) and libmesos (0.21.1) installed. +should have a JVM (Java 7 or higher) and libmesos (0.23.0) installed. ### Creating the Distribution .zip File (Optional) To create a distribution for installation you will need build tools installed. On Ubuntu this can be @@ -183,6 +187,25 @@ For example, monit can be configured wit assuming you set `-http_port=8081`. +## Security Considerations + +See [security.md](/documentation/latest/security/). + +## Configuring Resource Oversubscription + +**WARNING**: This feature is currently in alpha status. Do not use it in production clusters! +See [this document](configuration-reference.md#revocable-jobs) for more feature details. + +Set these scheduler flag to allow receiving revocable Mesos offers: + + -receive_revocable_resources=true + +Specify a tier configuration file path: + + -tier_config=path/to/tiers/config.json + +Example [tier configuration file](../src/test/resources/org/apache/aurora/scheduler/tiers-example.json). + ### Maintaining an Aurora Installation ### Monitoring @@ -202,6 +225,9 @@ constraints are arbitrary and available `dedicated` attribute. Aurora treats this specially, and only allows matching jobs to run on these machines, and will only schedule matching jobs on these machines. +See the [section](resources.md#resource-quota) about resource quotas to learn how quotas apply to +dedicated jobs. + ##### Syntax The dedicated attribute has semantic meaning. The format is `$role(/.*)?`. When a job is created, the scheduler requires that the `$role` component matches the `role` field in the job @@ -212,7 +238,7 @@ enforce this. ##### Example Consider the following slave command line: - mesos-slave --attributes="host:$HOST;rack:$RACK;dedicated:db_team/redis" ... + mesos-slave --attributes="dedicated:db_team/redis" ... And this job configuration: @@ -229,6 +255,19 @@ The job configuration is indicating that `dedicated:db_team/redis`. Additionally, Aurora will prevent any tasks that do _not_ have that constraint from running on those slaves. +## Best practices +### Diversity +Data centers are often organized with hierarchical failure domains. Common failure domains +include hosts, racks, rows, and PDUs. If you have this information available, it is wise to tag +the mesos-slave with them as +[attributes](https://mesos.apache.org/documentation/attributes-resources/). + +When it comes time to schedule jobs, Aurora will automatically spread them across the failure +domains as specified in the +[job configuration](configuration-reference.md#specifying-scheduling-constraints). + +Note: in virtualized environments like EC2, the only attribute that usually makes sense for this +purpose is `host`. ## Common problems So you've started your first cluster and are running into some issues? We've collected some common @@ -270,15 +309,18 @@ is the same as the one on the scheduler: -mesos_master_address=zk://$ZK_HOST:2181/mesos/master -### Tasks are stuck in `PENDING` forever - -#### Symptoms -The scheduler is registered, and (receiving offers](docs/monitoring.md#scheduler_resource_offers), -but tasks are perpetually shown as `PENDING - Constraint not satisfied: host`. - -#### Solution -Check that your slaves are configured with `host` and `rack` attributes. Aurora requires that -slaves are tagged with these two common failure domains to ensure that it can safely place tasks -such that jobs are resilient to failure. - -See our [vagrant example](examples/vagrant/upstart/mesos-slave.conf) for details. +## Changing Scheduler Quorum Size +Special care needs to be taken when changing the size of the Aurora scheduler quorum. +Since Aurora uses a Mesos replicated log, similar steps need to be followed as when +[changing the mesos quorum size](http://mesos.apache.org/documentation/latest/operational-guide). + +### Preparation +Increase [-native_log_quorum_size](storage-config.md#-native_log_quorum_size) on each +existing scheduler and restart them. When updating from 3 to 5 schedulers, the quorum size +would grow from 2 to 3. + +### Adding New Schedulers +Start the new schedulers with `-native_log_quorum_size` set to the new value. Failing to +first increase the quorum size on running schedulers can in some cases result in corruption +or truncating of the replicated log used by Aurora. In that case, see the documentation on +[recovering from backup](storage-config.md#recovering-from-a-scheduler-backup). Modified: aurora/site/source/documentation/latest/developing-aurora-client.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/developing-aurora-client.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/developing-aurora-client.md (original) +++ aurora/site/source/documentation/latest/developing-aurora-client.md Sat Dec 12 01:46:48 2015 @@ -19,7 +19,7 @@ Building and Testing the Client Building and testing the client code are both done using Pants. The relevant targets to know about are: - * Build a client executable: `./pants binary src/main/python/apache/aurora/client/cli:aurora` + * Build a client executable: `./pants binary src/main/python/apache/aurora/client:aurora` * Test client code: `./pants test src/test/python/apache/aurora/client/cli:all` Running/Debugging the Client @@ -30,7 +30,7 @@ To start a virtual cluster, you need to the aurora workspace. This will create a vagrant host named "devcluster", with a mesos master, a set of mesos slaves, and an aurora scheduler. -If you have changed you would like to test in your local cluster, you'll rebuild the client: +If you have a change you would like to test in your local cluster, you'll rebuild the client: vagrant ssh -c 'aurorabuild client' Modified: aurora/site/source/documentation/latest/developing-aurora-scheduler.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/developing-aurora-scheduler.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/developing-aurora-scheduler.md (original) +++ aurora/site/source/documentation/latest/developing-aurora-scheduler.md Sat Dec 12 01:46:48 2015 @@ -90,6 +90,25 @@ use the following commands to view and m bower update <library name> bower help +Faster Iteration in Vagrant +--------------------------- +The scheduler serves UI assets from the classpath. For production deployments this means the assets +are served from within a jar. However, for faster development iteration, the vagrant image is +configured to add `/vagrant/dist/resources/main` to the head of CLASSPATH. This path is configured +as a shared filesystem to the path on the host system where your Aurora repository lives. This means +that any updates to dist/resources/main in your checkout will be reflected immediately in the UI +served from within the vagrant image. + +The one caveat to this is that this path is under `dist` not `src`. This is because the assets must +be processed by gradle before they can be served. So, unfortunately, you cannot just save your local +changes and see them reflected in the UI, you must first run `./gradlew processResources`. This is +less than ideal, but better than having to restart the scheduler after every change. Additionally, +gradle makes this process somewhat easier with the use of the `--continuous` flag. If you run: +`./gradlew processResources --continuous` gradle will monitor the filesystem for changes and run the +task automatically as necessary. This doesn't quite provide hot-reload capabilities, but it does +allow for <5s from save to changes being visibile in the UI with no further action required on the +part of the developer. + Developing the Aurora Build System ================================== Modified: aurora/site/source/documentation/latest/monitoring.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/monitoring.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/monitoring.md (original) +++ aurora/site/source/documentation/latest/monitoring.md Sat Dec 12 01:46:48 2015 @@ -74,133 +74,108 @@ recommend you start with a strict value adjust thresholds as you see fit. Feel free to ask us if you would like to validate that your alerts and thresholds make sense. -#### `jvm_uptime_secs` +## Important stats + +### `jvm_uptime_secs` Type: integer counter -#### Description The number of seconds the JVM process has been running. Comes from [RuntimeMXBean#getUptime()](http://docs.oracle.com/javase/7/docs/api/java/lang/management/RuntimeMXBean.html#getUptime\(\)) -#### Alerting Detecting resets (decreasing values) on this stat will tell you that the scheduler is failing to stay alive. -#### Triage Look at the scheduler logs to identify the reason the scheduler is exiting. -#### `system_load_avg` +### `system_load_avg` Type: double gauge -#### Description The current load average of the system for the last minute. Comes from [OperatingSystemMXBean#getSystemLoadAverage()](http://docs.oracle.com/javase/7/docs/api/java/lang/management/OperatingSystemMXBean.html?is-external=true#getSystemLoadAverage\(\)). -#### Alerting A high sustained value suggests that the scheduler machine may be over-utilized. -#### Triage Use standard unix tools like `top` and `ps` to track down the offending process(es). -#### `process_cpu_cores_utilized` +### `process_cpu_cores_utilized` Type: double gauge -#### Description The current number of CPU cores in use by the JVM process. This should not exceed the number of logical CPU cores on the machine. Derived from [OperatingSystemMXBean#getProcessCpuTime()](http://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html) -#### Alerting A high sustained value indicates that the scheduler is overworked. Due to current internal design limitations, if this value is sustained at `1`, there is a good chance the scheduler is under water. -#### Triage There are two main inputs that tend to drive this figure: task scheduling attempts and status updates from Mesos. You may see activity in the scheduler logs to give an indication of where time is being spent. Beyond that, it really takes good familiarity with the code to effectively triage this. We suggest engaging with an Aurora developer. -#### `task_store_LOST` +### `task_store_LOST` Type: integer gauge -#### Description The number of tasks stored in the scheduler that are in the `LOST` state, and have been rescheduled. -#### Alerting If this value is increasing at a high rate, it is a sign of trouble. -#### Triage There are many sources of `LOST` tasks in Mesos: the scheduler, master, slave, and executor can all trigger this. The first step is to look in the scheduler logs for `LOST` to identify where the state changes are originating. -#### `scheduler_resource_offers` +### `scheduler_resource_offers` Type: integer counter -#### Description The number of resource offers that the scheduler has received. -#### Alerting For a healthy scheduler, this value must be increasing over time. -##### Triage Assuming the scheduler is up and otherwise healthy, you will want to check if the master thinks it is sending offers. You should also look at the master's web interface to see if it has a large number of outstanding offers that it is waiting to be returned. -#### `framework_registered` +### `framework_registered` Type: binary integer counter -#### Description Will be `1` for the leading scheduler that is registered with the Mesos master, `0` for passive schedulers, -#### Alerting A sustained period without a `1` (or where `sum() != 1`) warrants investigation. -#### Triage If there is no leading scheduler, look in the scheduler and master logs for why. If there are multiple schedulers claiming leadership, this suggests a split brain and warrants filing a critical bug. -#### `rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)` +### `rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)` Type: rate ratio of integer counters -#### Description This composes two counters to compute a windowed figure for the latency of replicated log writes. -#### Alerting A hike in this value suggests disk bandwidth contention. -#### Triage Look in scheduler logs for any reported oddness with saving to the replicated log. Also use standard tools like `vmstat` and `iotop` to identify whether the disk has become slow or over-utilized. We suggest using a dedicated disk for the replicated log to mitigate this. -#### `timed_out_tasks` +### `timed_out_tasks` Type: integer counter -#### Description Tracks the number of times the scheduler has given up while waiting (for `-transient_task_state_timeout`) to hear back about a task that is in a transient state (e.g. `ASSIGNED`, `KILLING`), and has moved to `LOST` before rescheduling. -#### Alerting This value is currently known to increase occasionally when the scheduler fails over ([AURORA-740](https://issues.apache.org/jira/browse/AURORA-740)). However, any large spike in this value warrants investigation. -#### Triage The scheduler will log when it times out a task. You should trace the task ID of the timed out task into the master, slave, and/or executors to determine where the message was dropped. -#### `http_500_responses_events` +### `http_500_responses_events` Type: integer counter -#### Description The total number of HTTP 500 status responses sent by the scheduler. Includes API and asset serving. -#### Alerting An increase warrants investigation. -#### Triage Look in scheduler logs to identify why the scheduler returned a 500, there should be a stack trace. Modified: aurora/site/source/documentation/latest/security.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/security.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/security.md (original) +++ aurora/site/source/documentation/latest/security.md Sat Dec 12 01:46:48 2015 @@ -144,8 +144,8 @@ You can then configure authorization usi To use Kerberos on the client-side you must build Kerberos-enabled client binaries. Do this with ``` -./pants binary src/main/python/apache/aurora/client/cli:kaurora -./pants binary src/main/python/apache/aurora/admin:kaurora_admin +./pants binary src/main/python/apache/aurora/kerberos:kaurora +./pants binary src/main/python/apache/aurora/kerberos:kaurora_admin ``` You must also configure each cluster where you've enabled Kerberos on the scheduler Modified: aurora/site/source/documentation/latest/sla.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/sla.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/sla.md (original) +++ aurora/site/source/documentation/latest/sla.md Sat Dec 12 01:46:48 2015 @@ -15,8 +15,9 @@ The primary goal of the feature is colle Agreements) metrics that defining a contractual relationship between the Aurora/Mesos platform and hosted services. -The Aurora SLA feature currently supports stat collection only for service (non-cron) -production jobs (`"production = True"` in your `.aurora` config). +The Aurora SLA feature is by default only enabled for service (non-cron) +production jobs (`"production = True"` in your `.aurora` config). It can be enabled for +non-production services via the scheduler command line flag `-sla_non_prod_metrics`. Counters that track SLA measurements are computed periodically within the scheduler. The individual instance metrics are refreshed every minute (configurable via @@ -173,4 +174,4 @@ unreasonable resource constraints) do no * The availability of Aurora SLA metrics is bound by the scheduler availability. * All metrics are calculated at a pre-defined interval (currently set at 1 minute). - Scheduler restarts may result in missed collections. \ No newline at end of file + Scheduler restarts may result in missed collections. Modified: aurora/site/source/documentation/latest/storage-config.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/storage-config.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/storage-config.md (original) +++ aurora/site/source/documentation/latest/storage-config.md Sat Dec 12 01:46:48 2015 @@ -99,10 +99,10 @@ accomplished by updating the following s * Set `-mesos_master_address` to a non-existent zk address. This will prevent scheduler from registering with Mesos. E.g.: `-mesos_master_address=zk://localhost:2181` * `-max_registration_delay` - set to sufficiently long interval to prevent registration timeout - and as a result scheduler suicide. E.g: `-max_registration_delay=360min` - * Make sure `-gc_executor_path` option is not set to prevent accidental task GC. This is - important as scheduler will attempt to reconcile the cluster state and will kill all tasks when - restarted with an empty Mesos replicated log. + and as a result scheduler suicide. E.g: `-max_registration_delay=360mins` + * Make sure `-reconciliation_initial_delay` option is set high enough (e.g.: `365days`) to + prevent accidental task GC. This is important as scheduler will attempt to reconcile the cluster + state and will kill all tasks when restarted with an empty Mesos replicated log. * Restart all schedulers @@ -112,7 +112,7 @@ Get rid of the corrupted files and re-in * Stop schedulers * Delete all files under `-native_log_file_path` on all schedulers -* Initialize Mesos replica's log file: `mesos-log initialize <-native_log_file_path>` +* Initialize Mesos replica's log file: `mesos-log initialize --path=<-native_log_file_path>` * Restart schedulers ### Restore from backup Modified: aurora/site/source/documentation/latest/test-resource-generation.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/test-resource-generation.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/test-resource-generation.md (original) +++ aurora/site/source/documentation/latest/test-resource-generation.md Sat Dec 12 01:46:48 2015 @@ -4,9 +4,8 @@ The Aurora source repository and distributions contain several [binary files](../src/test/resources/org/apache/thermos/root/checkpoints) to qualify the backwards-compatibility of thermos with checkpoint data. Since -thermos persists state to disk, to be read by other components (the GC executor -and the thermos observer), it is important that we have tests that prevent -regressions affecting the ability to parse previously-written data. +thermos persists state to disk, to be read by the thermos observer), it is important that we have +tests that prevent regressions affecting the ability to parse previously-written data. ## Generating test files The files included represent persisted checkpoints that exercise different Modified: aurora/site/source/documentation/latest/vagrant.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/latest/vagrant.md?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/source/documentation/latest/vagrant.md (original) +++ aurora/site/source/documentation/latest/vagrant.md Sat Dec 12 01:46:48 2015 @@ -43,7 +43,7 @@ Clone the Aurora repository To obtain the Aurora source distribution, clone its Git repository using the following command: - git clone http://git.apache.org/aurora.git + git clone git://git.apache.org/aurora.git Start the local cluster
