Added: aurora/site/source/documentation/0.12.0/hooks.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/hooks.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/hooks.md (added) +++ aurora/site/source/documentation/0.12.0/hooks.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,244 @@ +# Hooks for Aurora Client API + +- [Introduction](#introduction) +- [Hook Types](#hook-types) +- [Execution Order](#execution-order) +- [Hookable Methods](#hookable-methods) +- [Activating and Using Hooks](#activating-and-using-hooks) +- [.aurora Config File Settings](#aurora-config-file-settings) +- [Command Line](#command-line) +- [Hooks Protocol](#hooks-protocol) + - [pre_ Methods](#pre_-methods) + - [err_ Methods](#err_-methods) + - [post_ Methods](#post_-methods) +- [Generic Hooks](#generic-hooks) +- [Hooks Process Checklist](#hooks-process-checklist) + +## Introduction + +You can execute hook methods around Aurora API Client methods when they are called by the Aurora Command Line commands. + +Explaining how hooks work is a bit tricky because of some indirection about what they apply to. Basically, a hook is code that executes when a particular Aurora Client API method runs, letting you extend the method's actions. The hook executes on the client side, specifically on the machine executing Aurora commands. + +The catch is that hooks are associated with Aurora Client API methods, which users don't directly call. Instead, users call Aurora Command Line commands, which call Client API methods during their execution. Since which hooks run depend on which Client API methods get called, you will need to know which Command Line commands call which API methods. Later on, there is a table showing the various associations. + +**Terminology Note**: From now on, "method(s)" refer to Client API methods, and "command(s)" refer to Command Line commands. + +## Hook Types + +Hooks have three basic types, differing by when they run with respect to their associated method. + +`pre_<method_name>`: When its associated method is called, the `pre_` hook executes first, then the called method. If the `pre_` hook fails, the method never runs. Later code that expected the method to succeed may be affected by this, and result in terminating the Aurora client. + +Note that a `pre_` hook can error-trap internally so it does not +return `False`. Designers/contributors of new `pre_` hooks should +consider whether or not to error-trap them. You can error trap at the +highest level very generally and always pass the `pre_` hook by +returning `True`. For example: + +```python +def pre_create(...): + do_something() # if do_something fails with an exception, the create_job is not attempted! + return True + +# However... +def pre_create(...): + try: + do_something() # may cause exception + except Exception: # generic error trap will catch it + pass # and ignore the exception + return True # create_job will run in any case! +``` + +`post_<method_name>`: A `post_` hook executes after its associated method successfully finishes running. If it fails, the already executed method is unaffected. A `post_` hook's error is trapped, and any later operations are unaffected. + +`err_<method_name>`: Executes only when its associated method returns a status other than OK or throws an exception. If an `err_` hook fails, the already executed method is unaffected. An `err_` hook's error is trapped, and any later operations are unaffected. + +## Execution Order + +A command with `pre_`, `post_`, and `err_` hooks defined and activated for its called method executes in the following order when the method successfully executes: + +1. Command called +2. Command code executes +3. Method Called +4. `pre_` method hook runs +5. Method runs and successfully finishes +6. `post_` method hook runs +7. Command code executes +8. Command execution ends + +The following is what happens when, for the same command and hooks, the method associated with the command suffers an error and does not successfully finish executing: + +1. Command called +2. Command code executes +3. Method Called +4. `pre_` method hook runs +5. Method runs and fails +6. `err_` method hook runs +7. Command Code executes (if `err_` method does not end the command execution) +8. Command execution ends + +Note that the `post_` and `err_` hooks for the same method can never both run for a single execution of that method. + +## Hookable Methods + +You can associate `pre_`, `post_`, and `err_` hooks with the following methods. Since you do not directly interact with the methods, but rather the Aurora Command Line commands that call them, for each method we also list the command(s) that can call the method. Note that a different method or methods may be called by a command depending on how the command's other code executes. Similarly, multiple commands can call the same method. We also list the methods' argument signatures, which are used by their associated hooks. <a name="Chart"></a> + + Aurora Client API Method | Client API Method Argument Signature | Aurora Command Line Command + -------------------------| ------------------------------------- | --------------------------- + ```create_job``` | ```self```, ```config``` | ```job create```, <code>runtask + ```restart``` | ```self```, ```job_key```, ```shards```, ```update_config```, ```health_check_interval_seconds``` | ```job restart``` + ```kill_job``` | ```self```, ```job_key```, ```shards=None``` | ```job kill``` + ```start_cronjob``` | ```self```, ```job_key``` | ```cron start``` + ```start_job_update``` | ```self```, ```config```, ```instances=None``` | ```update start``` + +Some specific examples: + +* `pre_create_job` executes when a `create_job` method is called, and before the `create_job` method itself executes. + +* `post_cancel_update` executes after a `cancel_update` method has successfully finished running. + +* `err_kill_job` executes when the `kill_job` method is called, but doesn't successfully finish running. + +## Activating and Using Hooks + +By default, hooks are inactive. If you do not want to use hooks, you do not need to make any changes to your code. If you do want to use hooks, you will need to alter your `.aurora` config file to activate them both for the configuration as a whole as well as for individual `Job`s. And, of course, you will need to define in your config file what happens when a particular hook executes. + +## .aurora Config File Settings + +You can define a top-level `hooks` variable in any `.aurora` config file. `hooks` is a list of all objects that define hooks used by `Job`s defined in that config file. If you do not want to define any hooks for a configuration, `hooks` is optional. + + hooks = [Object_with_defined_hooks1, Object_with_defined_hooks2] + +Be careful when assembling a config file using `include` on multiple smaller config files. If there are multiple files that assign a value to `hooks`, only the last assignment made will stick. For example, if `x.aurora` has `hooks = [a, b, c]` and `y.aurora` has `hooks = [d, e, f]` and `z.aurora` has, in this order, `include x.aurora` and `include y.aurora`, the `hooks` value will be `[d, e, f]`. + +Also, for any `Job` that you want to use hooks with, its `Job` definition in the `.aurora` config file must set an `enable_hooks` flag to `True` (it defaults to `False`). By default, hooks are disabled and you must enable them for `Job`s of your choice. + +To summarize, to use hooks for a particular job, you must both activate hooks for your config file as a whole, and for that job. Activating hooks only for individual jobs won't work, nor will only activating hooks for your config file as a whole. You must also specify the hooks' defining object in the `hooks` variable. + +Recall that `.aurora` config files are written in Pystachio. So the following turns on hooks for production jobs at cluster1 and cluster2, but leaves them off for similar jobs with a defined user role. Of course, you also need to list the objects that define the hooks in your config file's `hooks` variable. + +```python +jobs = [ + Job(enable_hooks = True, cluster = c, env = 'prod') for c in ('cluster1', 'cluster2') + ] +jobs.extend( + Job(cluster = c, env = 'prod', role = getpass.getuser()) for c in ('cluster1', 'cluster2')) + # Hooks disabled for these jobs +``` + +## Command Line + +All Aurora Command Line commands now accept an `.aurora` config file as an optional parameter (some, of course, accept it as a required parameter). Whenever a command has a `.aurora` file parameter, any hooks specified and activated in the `.aurora` file can be used. For example: + + aurora job restart cluster1/role/env/app myapp.aurora + +The command activates any hooks specified and activated in `myapp.aurora`. For the `restart` command, that is the only thing the `myapp.aurora` parameter does. So, if the command was the following, since there is no `.aurora` config file to specify any hooks, no hooks on the `restart` command can run. + + aurora job restart cluster1/role/env/app + +## Hooks Protocol + +Any object defined in the `.aurora` config file can define hook methods. You should define your hook methods within a class, and then use the class name as a value in the `hooks` list in your config file. + +Note that you can define other methods in the class that its hook methods can call; all the logic of a hook does not have to be in its definition. + +The following example defines a class containing a `pre_kill_job` hook definition that calls another method defined in the class. + +```python +# Defines a method pre_kill_job +class KillConfirmer(object): + def confirm(self, msg): + return raw_input(msg).lower() == 'yes' + + def pre_kill_job(self, job_key, shards=None): + shards = ('shards %s' % shards) if shards is not None else 'all shards' + return self.confirm('Are you sure you want to kill %s (%s)? (yes/no): ' + % (job_key, shards)) +``` + +### pre_ Methods + +`pre_` methods have the signature: + + pre_<API method name>(self, <associated method's signature>) + +`pre_` methods have the same signature as their associated method, with the addition of `self` as the first parameter. See the [chart](#Chart) above for the mapping of parameters to methods. When writing `pre_` methods, you can use the `*` and `**` syntax to designate that all unspecified parameters are passed in a list to the `*`ed variable and all named parameters with values are passed as name/value pairs to the `**`ed variable. + +If this method returns False, the API command call aborts. + +### err_ Methods + +`err_` methods have the signature: + + err_<API method name>(self, exc, <associated method's signature>) + +`err_` methods have the same signature as their associated method, with the addition of a first parameter `self` and a second parameter `exc`. `exc` is either a result with responseCode other than `ResponseCode.OK` or an `Exception`. See the [chart](#Chart) above for the mapping of parameters to methods. When writing `err`_ methods, you can use the `*` and `**` syntax to designate that all unspecified parameters are passed in a list to the `*`ed variable and all named parameters with values are passed as name/value pairs to the `**`ed variable. + +`err_` method return codes are ignored. + +### post_ Methods + +`post_` methods have the signature: + + post_<API method name>(self, result, <associated method signature>) + +`post_` method parameters are `self`, then `result`, followed by the same parameter signature as their associated method. `result` is the result of the associated method call. See the [chart](#chart) above for the mapping of parameters to methods. When writing `post_` methods, you can use the `*` and `**` syntax to designate that all unspecified arguments are passed in a list to the `*`ed parameter and all unspecified named arguments with values are passed as name/value pairs to the `**`ed parameter. + +`post_` method return codes are ignored. + +## Generic Hooks + +There are seven Aurora API Methods which any of the three hook types can attach to. Thus, there are 21 possible hook/method combinations for a single `.aurora` config file. Say that you define `pre_` and `post_` hooks for the `restart` method. That leaves 19 undefined hook/method combinations; `err_restart` and the 3 `pre_`, `post_`, and `err_` hooks for each of the other 6 hookable methods. You can define what happens when any of these otherwise undefined 19 hooks execute via a generic hook, whose signature is: + +```python +generic_hook(self, hook_config, event, method_name, result_or_err, args*, kw**) +``` + +where: + +* `hook_config` is a named tuple of `config` (the Pystashio `config` object) and `job_key`. + +* `event` is one of `pre`, `err`, or `post`, indicating which type of hook the genetic hook is standing in for. For example, assume no specific hooks were defined for the `restart` API command. If `generic_hook` is defined and activated, and `restart` is called, `generic_hook` will effectively run as `pre_restart`, `post_restart`, and `err_restart`. You can use a selection statement on this value so that `generic_hook` will act differently based on whether it is standing in for a `pre_`, `post_`, or `err_` hook. + +* `method_name` is the Client API method name whose execution is causing this execution of the `generic_hook`. + +* `args*`, `kw**` are the API method arguments and keyword arguments respectively. +* `result_or_err` is a tri-state parameter taking one of these three values: + 1. None for `pre_`hooks + 2. `result` for `post_` nooks + 3. `exc` for `err_` hooks + +Example: + +```python +# Overrides the standard do-nothing generic_hook by adding a log writing operation. +from twitter.common import log + class Logger(object): + '''Adds to the log every time a hookable API method is called''' + def generic_hook(self, hook_config, event, method_name, result_or_err, *args, **kw) + log.info('%s: %s_%s of %s' + % (self.__class__.__name__, event, method_name, hook_config.job_key)) +``` + +## Hooks Process Checklist + +1. In your `.aurora` config file, add a `hooks` variable. Note that you may want to define a `.aurora` file only for hook definitions and then include this file in multiple other config files that you want to use the same hooks. + +```python +hooks = [] +``` + +2. In the `hooks` variable, list all objects that define hooks used by `Job`s defined in this config: + +```python +hooks = [Object_hook_definer1, Object_hook_definer2] +``` + +3. For each job that uses hooks in this config file, add `enable_hooks = True` to the `Job` definition. Note that this is necessary even if you only want to use the generic hook. + +4. Write your `pre_`, `post_`, and `err_` hook definitions as part of an object definition in your `.aurora` config file. + +5. If desired, write your `generic_hook` definition as part of an object definition in your `.aurora` config file. Remember, the object must be listed as a member of `hooks`. + +6. If your Aurora command line command does not otherwise take an `.aurora` config file argument, add the appropriate `.aurora` file as an argument in order to define and activate the configuration's hooks.
Added: aurora/site/source/documentation/0.12.0/images/CPUavailability.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/CPUavailability.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/CPUavailability.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/CompletedTasks.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/CompletedTasks.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/CompletedTasks.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/HelloWorldJob.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/HelloWorldJob.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/HelloWorldJob.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/RoleJobs.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/RoleJobs.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/RoleJobs.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/RunningJob.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/RunningJob.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/RunningJob.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/ScheduledJobs.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/ScheduledJobs.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/ScheduledJobs.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/TaskBreakdown.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/TaskBreakdown.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/TaskBreakdown.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/aurora_hierarchy.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/aurora_hierarchy.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/aurora_hierarchy.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/aurora_logo.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/aurora_logo.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/aurora_logo.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/components.odg URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/components.odg?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/components.odg ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/components.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/components.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/components.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/debug-client-test.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/debug-client-test.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/debug-client-test.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/debugging-client-test.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/debugging-client-test.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/debugging-client-test.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/killedtask.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/killedtask.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/killedtask.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/lifeofatask.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/lifeofatask.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/lifeofatask.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/02_28_2015_apache_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/02_28_2015_apache_aurora_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/02_28_2015_apache_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/08_21_2014_past_present_future_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/08_21_2014_past_present_future_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/08_21_2014_past_present_future_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/runningtask.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/runningtask.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/runningtask.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/stderr.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/stderr.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/stderr.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/stdout.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/stdout.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/stdout.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/images/storage_hierarchy.png URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/images/storage_hierarchy.png?rev=1733548&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/source/documentation/0.12.0/images/storage_hierarchy.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: aurora/site/source/documentation/0.12.0/index.html.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/index.html.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/index.html.md (added) +++ aurora/site/source/documentation/0.12.0/index.html.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,44 @@ +## Introduction +Apache Aurora is a service scheduler that runs on top of Apache Mesos, enabling you to run long-running services that take advantage of Apache Mesos' scalability, fault-tolerance, and resource isolation. This documentation has been organized into sections with three audiences in mind: + + * Users: General information about the project and to learn how to run an Aurora job. + * Operators: For those that wish to manage and fine-tune an Aurora cluster. + * Developers: All the information you need to start modifying Aurora and contributing back to the project. + +We encourage you to ask questions on the [Aurora user list](http://aurora.apache.org/community/) or the `#aurora` IRC channel on `irc.freenode.net`. + +## Users + * [Install Aurora on virtual machines on your private machine](/documentation/0.12.0/vagrant/) + * [Hello World Tutorial](/documentation/0.12.0/tutorial/) + * [User Guide](/documentation/0.12.0/user-guide/) + * [Configuration Tutorial](/documentation/0.12.0/configuration-tutorial/) + * [Aurora + Thermos Reference](/documentation/0.12.0/configuration-reference/) + * [Command Line Client](/documentation/0.12.0/client-commands/) + * [Client cluster configuration](/documentation/0.12.0/client-cluster-configuration/) + * [Cron Jobs](/documentation/0.12.0/cron-jobs/) + +## Operators + * [Installation](/documentation/0.12.0/installing/) + * [Deployment and cluster configuration](/documentation/0.12.0/deploying-aurora-scheduler/) + * [Security](/documentation/0.12.0/security/) + * [Monitoring](/documentation/0.12.0/monitoring/) + * [Hooks for Aurora Client API](/documentation/0.12.0/hooks/) + * [Scheduler Storage](/documentation/0.12.0/storage/) + * [Scheduler Storage and Maintenance](/documentation/0.12.0/storage-config/) + * [SLA Measurement](/documentation/0.12.0/sla/) + * [Resource Isolation and Sizing](/documentation/0.12.0/resources/) + +## Developers + * [Contributing to the project](contributing/) + * [Developing the Aurora Scheduler](/documentation/0.12.0/developing-aurora-scheduler/) + * [Developing the Aurora Client](/documentation/0.12.0/developing-aurora-client/) + * [Committers Guide](/documentation/0.12.0/committers/) + * [Design Documents](/documentation/0.12.0/design-documents/) + * [Deprecation Guide](/documentation/0.12.0/thrift-deprecation/) + * [Build System](/documentation/0.12.0/build-system/) + * [Generating test resources](/documentation/0.12.0/test-resource-generation/) + + +## Additional Resources + * [Tools integrating with Aurora](/documentation/0.12.0/tools/) + * [Presentation videos and slides](/documentation/0.12.0/presentations/) Added: aurora/site/source/documentation/0.12.0/installing.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/installing.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/installing.md (added) +++ aurora/site/source/documentation/0.12.0/installing.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,279 @@ +# Installing Aurora + +- [Components](#components) + - [Machine profiles](#machine-profiles) + - [Coordinator](#coordinator) + - [Worker](#worker) + - [Client](#client) +- [Getting Aurora](#getting-aurora) + - [Building your own binary packages](#building-your-own-binary-packages) + - [RPMs](#rpms) +- [Installing the scheduler](#installing-the-scheduler) + - [Ubuntu Trusty](#ubuntu-trusty) + - [CentOS 7](#centos-7) + - [Finalizing](#finalizing) + - [Configuration](#configuration) +- [Installing worker components](#installing-worker-components) + - [Ubuntu Trusty](#ubuntu-trusty-1) + - [CentOS 7](#centos-7-1) + - [Configuration](#configuration-1) +- [Installing the client](#installing-the-client) + - [Ubuntu Trusty](#ubuntu-trusty-2) + - [CentOS 7](#centos-7-2) + - [Configuration](#configuration-2) +- [See also](#see-also) +- [Installing Mesos](#installing-mesos) + - [Mesos on Ubuntu Trusty](#mesos-on-ubuntu-trusty) + - [Mesos on CentOS 7](#mesos-on-centos-7) + +## Components +Before installing Aurora, it's important to have an understanding of the components that make up +a functioning Aurora cluster. + + + +* **Aurora scheduler** + The scheduler will be your primary interface to the work you run in your cluster. You will + instruct it to run jobs, and it will manage them in Mesos for you. You will also frequently use + the scheduler's web interface as a heads-up display for what's running in your cluster. + +* **Aurora client** + The client (`aurora` command) is a command line tool that exposes primitives that you can use to + interact with the scheduler. + + Aurora also provides an admin client (`aurora_admin` command) that contains commands built for + cluster administrators. You can use this tool to do things like manage user quotas and manage + graceful maintenance on machines in cluster. + +* **Aurora executor** + The executor (a.k.a. Thermos executor) is responsible for carrying out the workloads described in + the Aurora DSL (`.aurora` files). The executor is what actually executes user processes. It will + also perform health checking of tasks and register tasks in ZooKeeper for the purposes of dynamic + service discovery. You can find lots more detail on the executor and Thermos in the + [user guide](/documentation/0.12.0/user-guide/). + +* **Aurora observer** + The observer provides browser-based access to the status of individual tasks executing on worker + machines. It gives insight into the processes executing, and facilitates browsing of task sandbox + directories. + +* **ZooKeeper** + [ZooKeeper](http://zookeeper.apache.org) is a distributed consensus system. In an Aurora cluster + it is used for reliable election of the leading Aurora scheduler and Mesos master. + +* **Mesos master** + The master is responsible for tracking worker machines and performing accounting of their + resources. The scheduler interfaces with the master to control the cluster. + +* **Mesos agent** + The agent receives work assigned by the scheduler and executes them. It interfaces with Linux + isolation systems like cgroups, namespaces and Docker to manage the resource consumption of tasks. + When a user task is launched, the agent will launch the executor (in the context of a Linux cgroup + or Docker container depending upon the environment), which will in turn fork user processes. + +## Machine profiles +Given that many of these components communicate over the network, there are numerous ways you could +assemble them to create an Aurora cluster. The simplest way is to think in terms of three machine +profiles: + +### Coordinator +**Components**: ZooKeeper, Aurora scheduler, Mesos master + +A small number of machines (typically 3 or 5) responsible for cluster orchestration. In most cases +it is fine to co-locate these components in anything but very large clusters (> 1000 machines). +Beyond that point, operators will likely want to manage these services on separate machines. + +In practice, 5 coordinators have been shown to reliably manage clusters with tens of thousands of +machines. + + +### Worker +**Components**: Aurora executor, Aurora observer, Mesos agent + +The bulk of the cluster, where services will actually run. + +### Client +**Components**: Aurora client, Aurora admin client + +Any machines that users submit jobs from. + +## Getting Aurora +Source and binary distributions can be found on our +[downloads](https://aurora.apache.org/downloads/) page. Installing from binary packages is +recommended for most. + +### Building your own binary packages +Our package build toolchain makes it easy to build your own packages if you would like. See the +[instructions](https://github.com/apache/aurora-packaging) to learn how. + +### RPMs +We currently have work in progress to provide official RPMs. As of this writing, the suggested way +to get RPMs is to [build them](#building-your-own-binary-packages). + +We do have unofficial experimental RPMs available for testing purposes. + +**Use these RPMs at your own risk, they are not officially released under the ASF guidelines.** + + echo '[apache-aurora-wfarner] + name=Apache Aurora distribution maintained by wfarner + baseurl=http://people.apache.org/~wfarner/aurora/distributions/0.9.0/rpm/centos-7/x86_64/ + gpgcheck = 0' | sudo tee /etc/yum.repos.d/apache-aurora-wfarner.repo > /dev/null + + +## Installing the scheduler +### Ubuntu Trusty + +1. Install Mesos + Skip down to [install mesos](#mesos-on-ubuntu-trusty), then run: + + sudo start mesos-master + +2. Install ZooKeeper + + sudo apt-get install -y zookeeperd + +3. Install the Aurora scheduler + + wget -c https://apache.bintray.com/aurora/aurora-scheduler_0.10.0-1_amd64.deb + sudo dpkg -i aurora-scheduler_0.10.0-1_amd64.deb + +### CentOS 7 + +1. Install Mesos + Skip down to [install mesos](#mesos-on-centos-7), then run: + + sudo systemctl start mesos-master + +2. Install ZooKeeper + + sudo rpm -Uvh https://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm + sudo yum install -y java-1.8.0-openjdk-headless zookeeper-server + + sudo service zookeeper-server init + sudo systemctl start zookeeper-server + +3. Install the Aurora scheduler + If you haven't already, read the section on [how to get Aurora RPMs](#rpms). + + # Note: for older Aurora RPM versions, this may be called 'aurora'. + sudo yum install -y aurora-scheduler + +Note: if you are using the unreleased 0.9.0 RPM, you will need to edit `/etc/sysconfig/aurora`: +Change +`-mesos_master_address='zk://127.0.0.1:2181/mesos/master'` +To +`-mesos_master_address='zk://127.0.0.1:2181/mesos'` +And +`-native_log_file_path='/var/lib/aurora/db'` +To +`-native_log_file_path='/var/lib/aurora/scheduler/db'` + +### Finalizing +By default, the scheduler will start in an uninitialized mode. This is because external +coordination is necessary to be certain operator error does not result in a quorum of schedulers +starting up and believing their databases are empty when in fact they should be re-joining a +cluster. + +Because of this, a fresh install of the scheduler will need intervention to start up. First, +stop the scheduler service. +Ubuntu: `sudo stop aurora-scheduler` +CentOS: `sudo systemctl stop aurora` + +Now initialize the database: + + sudo -u aurora mkdir -p /var/lib/aurora/scheduler/db + sudo -u aurora mesos-log initialize --path=/var/lib/aurora/scheduler/db + +Now you can start the scheduler back up. +Ubuntu: `sudo start aurora-scheduler` +CentOS: `sudo systemctl start aurora` + +### Configuration +For more detail on this topic, see the dedicated page on +[deploying the scheduler](/documentation/0.12.0/deploying-aurora-scheduler/) + + +## Installing worker components +### Ubuntu Trusty + +1. Install Mesos + Skip down to [install mesos](#mesos-on-ubuntu-trusty), then run: + + sudo start mesos-slave + +2. Install Aurora executor and observer + + wget -c https://apache.bintray.com/aurora/aurora-executor_0.10.0-1_amd64.deb + sudo dpkg -i aurora-executor_0.10.0-1_amd64.deb + +### CentOS 7 + +1. Install Mesos + Skip down to [install mesos](#mesos-on-centos-7), then run: + + sudo systemctl start mesos-slave + +2. Install Aurora executor and observer + If you haven't already, read the section on [how to get Aurora RPMs](#rpms). + + # Note: for older Aurora RPM versions, this may be called 'aurora-thermos'. + sudo yum install -y aurora-executor + +### Configuration +The executor and observer typically do not require much configuration. Command line arguments can +be passed to the executor using a command line argument on the scheduler. + +## Installing the client +### Ubuntu Trusty + + sudo apt-get install -y python2.7 wget + + wget https://apache.bintray.com/aurora/aurora-tools_0.10.0-1_amd64.deb + sudo dpkg -i aurora-tools_0.10.0-1_amd64.deb + +### CentOS 7 +If you haven't already, read the section on [how to get Aurora RPMs](#rpms). + + # Note: for older Aurora RPM versions, this may be called 'aurora-client'. + sudo yum install -y aurora-tools + +### Configuration +Client configuration lives in a json file that describes the clusters available and how to reach +them. By default this file is at `/etc/aurora/clusters.json`. + +Jobs may be submitted to the scheduler using the client, and are described with +[job configurations](/documentation/0.12.0/configuration-reference/) expressed in `.aurora` files. Typically you will +maintain a single job configuration file to describe one or more deployment environments (e.g. +dev, test, prod) for a production job. + +## See also +We have other docs that you will find useful once you have your cluster up and running: + +- [Monitor](/documentation/0.12.0/monitoring/) your cluster +- Enable scheduler [security](/documentation/0.12.0/security/) +- View job SLA [statistics](/documentation/0.12.0/sla/) +- Understand the internals of the scheduler's [storage](/documentation/0.12.0/storage/) + +## Installing Mesos +Mesos uses a single package for the Mesos master and slave. As a result, the package dependencies +are identical for both. + +### Mesos on Ubuntu Trusty + + sudo apt-get update + sudo apt-get install -y software-properties-common + sudo add-apt-repository ppa:openjdk-r/ppa -y + sudo apt-get update + + sudo apt-get install -y wget libsvn1 libcurl3 openjdk-8-jre-headless + + # NOTE: This appears to be a missing dependency of the mesos deb package. + sudo apt-get install -y libcurl4-nss-dev + + wget -c http://downloads.mesosphere.io/master/ubuntu/14.04/mesos_0.23.0-1.0.ubuntu1404_amd64.deb + sudo dpkg -i mesos_0.23.0-1.0.ubuntu1404_amd64.deb + +### Mesos on CentOS 7 + + sudo rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm + sudo yum install -y mesos-0.22.0 Added: aurora/site/source/documentation/0.12.0/monitoring.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/monitoring.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/monitoring.md (added) +++ aurora/site/source/documentation/0.12.0/monitoring.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,181 @@ +# Monitoring your Aurora cluster + +Before you start running important services in your Aurora cluster, it's important to set up +monitoring and alerting of Aurora itself. Most of your monitoring can be against the scheduler, +since it will give you a global view of what's going on. + +## Reading stats +The scheduler exposes a *lot* of instrumentation data via its HTTP interface. You can get a quick +peek at the first few of these in our vagrant image: + + $ vagrant ssh -c 'curl -s localhost:8081/vars | head' + async_tasks_completed 1004 + attribute_store_fetch_all_events 15 + attribute_store_fetch_all_events_per_sec 0.0 + attribute_store_fetch_all_nanos_per_event 0.0 + attribute_store_fetch_all_nanos_total 3048285 + attribute_store_fetch_all_nanos_total_per_sec 0.0 + attribute_store_fetch_one_events 3391 + attribute_store_fetch_one_events_per_sec 0.0 + attribute_store_fetch_one_nanos_per_event 0.0 + attribute_store_fetch_one_nanos_total 454690753 + +These values are served as `Content-Type: text/plain`, with each line containing a space-separated metric +name and value. Values may be integers, doubles, or strings (note: strings are static, others +may be dynamic). + +If your monitoring infrastructure prefers JSON, the scheduler exports that as well: + + $ vagrant ssh -c 'curl -s localhost:8081/vars.json | python -mjson.tool | head' + { + "async_tasks_completed": 1009, + "attribute_store_fetch_all_events": 15, + "attribute_store_fetch_all_events_per_sec": 0.0, + "attribute_store_fetch_all_nanos_per_event": 0.0, + "attribute_store_fetch_all_nanos_total": 3048285, + "attribute_store_fetch_all_nanos_total_per_sec": 0.0, + "attribute_store_fetch_one_events": 3409, + "attribute_store_fetch_one_events_per_sec": 0.0, + "attribute_store_fetch_one_nanos_per_event": 0.0, + +This will be the same data as above, served with `Content-Type: application/json`. + +## Viewing live stat samples on the scheduler +The scheduler uses the Twitter commons stats library, which keeps an internal time-series database +of exported variables - nearly everything in `/vars` is available for instant graphing. This is +useful for debugging, but is not a replacement for an external monitoring system. + +You can view these graphs on a scheduler at `/graphview`. It supports some composition and +aggregation of values, which can be invaluable when triaging a problem. For example, if you have +the scheduler running in vagrant, check out these links: +[simple graph](http://192.168.33.7:8081/graphview?query=jvm_uptime_secs) +[complex composition](http://192.168.33.7:8081/graphview?query=rate\(scheduler_log_native_append_nanos_total\)%2Frate\(scheduler_log_native_append_events\)%2F1e6) + +### Counters and gauges +Among numeric stats, there are two fundamental types of stats exported: _counters_ and _gauges_. +Counters are guaranteed to be monotonically-increasing for the lifetime of a process, while gauges +may decrease in value. Aurora uses counters to represent things like the number of times an event +has occurred, and gauges to capture things like the current length of a queue. Counters are a +natural fit for accurate composition into [rate ratios](http://en.wikipedia.org/wiki/Rate_ratio) +(useful for sample-resistant latency calculation), while gauges are not. + +# Alerting + +## Quickstart +If you are looking for just bare-minimum alerting to get something in place quickly, set up alerting +on `framework_registered` and `task_store_LOST`. These will give you a decent picture of overall +health. + +## A note on thresholds +One of the most difficult things in monitoring is choosing alert thresholds. With many of these +stats, there is no value we can offer as a threshold that will be guaranteed to work for you. It +will depend on the size of your cluster, number of jobs, churn of tasks in the cluster, etc. We +recommend you start with a strict value after viewing a small amount of collected data, and then +adjust thresholds as you see fit. Feel free to ask us if you would like to validate that your alerts +and thresholds make sense. + +## Important stats + +### `jvm_uptime_secs` +Type: integer counter + +The number of seconds the JVM process has been running. Comes from +[RuntimeMXBean#getUptime()](http://docs.oracle.com/javase/7/docs/api/java/lang/management/RuntimeMXBean.html#getUptime\(\)) + +Detecting resets (decreasing values) on this stat will tell you that the scheduler is failing to +stay alive. + +Look at the scheduler logs to identify the reason the scheduler is exiting. + +### `system_load_avg` +Type: double gauge + +The current load average of the system for the last minute. Comes from +[OperatingSystemMXBean#getSystemLoadAverage()](http://docs.oracle.com/javase/7/docs/api/java/lang/management/OperatingSystemMXBean.html?is-external=true#getSystemLoadAverage\(\)). + +A high sustained value suggests that the scheduler machine may be over-utilized. + +Use standard unix tools like `top` and `ps` to track down the offending process(es). + +### `process_cpu_cores_utilized` +Type: double gauge + +The current number of CPU cores in use by the JVM process. This should not exceed the number of +logical CPU cores on the machine. Derived from +[OperatingSystemMXBean#getProcessCpuTime()](http://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html) + +A high sustained value indicates that the scheduler is overworked. Due to current internal design +limitations, if this value is sustained at `1`, there is a good chance the scheduler is under water. + +There are two main inputs that tend to drive this figure: task scheduling attempts and status +updates from Mesos. You may see activity in the scheduler logs to give an indication of where +time is being spent. Beyond that, it really takes good familiarity with the code to effectively +triage this. We suggest engaging with an Aurora developer. + +### `task_store_LOST` +Type: integer gauge + +The number of tasks stored in the scheduler that are in the `LOST` state, and have been rescheduled. + +If this value is increasing at a high rate, it is a sign of trouble. + +There are many sources of `LOST` tasks in Mesos: the scheduler, master, slave, and executor can all +trigger this. The first step is to look in the scheduler logs for `LOST` to identify where the +state changes are originating. + +### `scheduler_resource_offers` +Type: integer counter + +The number of resource offers that the scheduler has received. + +For a healthy scheduler, this value must be increasing over time. + +Assuming the scheduler is up and otherwise healthy, you will want to check if the master thinks it +is sending offers. You should also look at the master's web interface to see if it has a large +number of outstanding offers that it is waiting to be returned. + +### `framework_registered` +Type: binary integer counter + +Will be `1` for the leading scheduler that is registered with the Mesos master, `0` for passive +schedulers, + +A sustained period without a `1` (or where `sum() != 1`) warrants investigation. + +If there is no leading scheduler, look in the scheduler and master logs for why. If there are +multiple schedulers claiming leadership, this suggests a split brain and warrants filing a critical +bug. + +### `rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)` +Type: rate ratio of integer counters + +This composes two counters to compute a windowed figure for the latency of replicated log writes. + +A hike in this value suggests disk bandwidth contention. + +Look in scheduler logs for any reported oddness with saving to the replicated log. Also use +standard tools like `vmstat` and `iotop` to identify whether the disk has become slow or +over-utilized. We suggest using a dedicated disk for the replicated log to mitigate this. + +### `timed_out_tasks` +Type: integer counter + +Tracks the number of times the scheduler has given up while waiting +(for `-transient_task_state_timeout`) to hear back about a task that is in a transient state +(e.g. `ASSIGNED`, `KILLING`), and has moved to `LOST` before rescheduling. + +This value is currently known to increase occasionally when the scheduler fails over +([AURORA-740](https://issues.apache.org/jira/browse/AURORA-740)). However, any large spike in this +value warrants investigation. + +The scheduler will log when it times out a task. You should trace the task ID of the timed out +task into the master, slave, and/or executors to determine where the message was dropped. + +### `http_500_responses_events` +Type: integer counter + +The total number of HTTP 500 status responses sent by the scheduler. Includes API and asset serving. + +An increase warrants investigation. + +Look in scheduler logs to identify why the scheduler returned a 500, there should be a stack trace. Added: aurora/site/source/documentation/0.12.0/presentations.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/presentations.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/presentations.md (added) +++ aurora/site/source/documentation/0.12.0/presentations.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,80 @@ +# Apache Aurora Presentations +Video and slides from presentations and panel discussions about Apache Aurora. + +_(Listed in date descending order)_ + +<table> + + <tr> + <td><img src="/documentation/0.12.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png" alt="Mesos and Aurora on a Small Scale Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=q5iIqhaCJ_o";>Mesos & Aurora on a Small Scale (Video)</a></strong> + <p>Presented by Florian Pfeiffer</p> + <p>October 8, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon-europe";>#MesosCon Europe 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png" alt="SLA Aware Maintenance for Operators Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=tZ0-SISvCis";>SLA Aware Maintenance for Operators (Video)</a></strong> + <p>Presented by Joe Smith</p> + <p>October 8, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon-europe";>#MesosCon Europe 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png" alt="Shipping Code with Aurora Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=y1hi7K1lPkk";>Shipping Code with Aurora (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>August 20, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon";>#MesosCon 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png" alt="Twitter Production Scale Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=nNrh-gdu9m4";>Twitterâs Production Scale: Mesos and Aurora Operations (Video)</a></strong> + <p>Presented by Joe Smith</p> + <p>August 20, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon";>#MesosCon 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png" alt="From Monolith to Microservices with Aurora Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=yXkOgnyK4Hw";>>From Monolith to Microservices w/ Aurora (Video)</a></strong> + <p>Presented by Thanos Baskous, Tony Dong, Dobromir Montauk</p> + <p>April 30, 2015 at <a href="http://www.meetup.com/Bay-Area-Apache-Aurora-Users-Group/events/221219480/";>Bay Area Apache Aurora Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png" alt="Aurora + Mesos in Practice at Twitter Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=1XYJGX_qZVU";>Aurora + Mesos in Practice at Twitter (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>March 07, 2015 at <a href="http://www.bigeng.io/aurora-mesos-in-practice-at-twitter";>Bigcommerce TechTalk</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/02_28_2015_apache_aurora_thumb.png" alt="Apache Auroraã®å§ããã Slideshow Thumbnail" /></td> + <td><strong><a href="http://www.slideshare.net/zembutsu/apache-aurora-introduction-and-tutorial-osc15tk";>Apache Auroraã®å§ããã (Slides)</a></strong> + <p>Presented by Masahito Zembutsu</p> + <p>February 28, 2015 at <a href="http://www.ospn.jp/osc2015-spring/";>Open Source Conference 2015 Tokyo Spring</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png" alt="Apache Aurora Adopters Panel Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=2Jsj0zFdRlg";>Apache Aurora Adopters Panel (Video)</a></strong> + <p>Panelists Ben Staffin, Josh Adams, Bill Farner, Berk Demir</p> + <p>February 19, 2015 at <a href="http://www.meetup.com/Bay-Area-Mesos-User-Group/events/220279080/";>Bay Area Mesos Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png" alt="Operating Apache Aurora and Mesos at Twitter Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=E4lxX6epM_U";>Operating Apache Aurora and Mesos at Twitter (Video)</a></strong> + <p>Presented by Joe Smith</p> + <p>February 19, 2015 at <a href="http://www.meetup.com/Bay-Area-Mesos-User-Group/events/220279080/";>Bay Area Mesos Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png" alt="Apache Aurora and Mesos at TellApart" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=ZZXtXLvTXAE";>Apache Aurora and Mesos at TellApart (Video)</a></strong> + <p>Presented by Steve Niemitz</p> + <p>February 19, 2015 at <a href="http://www.meetup.com/Bay-Area-Mesos-User-Group/events/220279080/";>Bay Area Mesos Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/08_21_2014_past_present_future_thumb.png" alt="Past, Present, and Future of the Aurora Scheduler Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=Dsc5CPhKs4o";>Past, Present, and Future of the Aurora Scheduler (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>August 21, 2014 at <a href="http://events.linuxfoundation.org/events/archive/2014/mesoscon";>#MesosCon 2014</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.12.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png" alt="Introduction to Apache Aurora Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=asd_h6VzaJc";>Introduction to Apache Aurora (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>March 25, 2014 at <a href="https://www.eventbrite.com/e/aurora-and-mesosframeworksmeetup-tickets-10850994617";>Aurora and Mesos Frameworks Meetup</a></p></td> + </tr> +</table> Added: aurora/site/source/documentation/0.12.0/resources.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/resources.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/resources.md (added) +++ aurora/site/source/documentation/0.12.0/resources.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,164 @@ +Resources and Sizing +============================= + +- [Introduction](#introduction) +- [CPU Isolation](#cpu-isolation) +- [CPU Sizing](#cpu-sizing) +- [Memory Isolation](#memory-isolation) +- [Memory Sizing](#memory-sizing) +- [Disk Space](#disk-space) +- [Disk Space Sizing](#disk-space-sizing) +- [Other Resources](#other-resources) +- [Resource Quota](#resource-quota) +- [Task Preemption](#task-preemption) + +## Introduction + +Aurora is a multi-tenant system; a single software instance runs on a +server, serving multiple clients/tenants. To share resources among +tenants, it implements isolation of: + +* CPU +* memory +* disk space + +CPU is a soft limit, and handled differently from memory and disk space. +Too low a CPU value results in throttling your application and +slowing it down. Memory and disk space are both hard limits; when your +application goes over these values, it's killed. + +Let's look at each resource type in more detail: + +## CPU Isolation + +Mesos uses a quota based CPU scheduler (the *Completely Fair Scheduler*) +to provide consistent and predictable performance. This is effectively +a guarantee of resources -- you receive at least what you requested, but +also no more than you've requested. + +The scheduler gives applications a CPU quota for every 100 ms interval. +When an application uses its quota for an interval, it is throttled for +the rest of the 100 ms. Usage resets for each interval and unused +quota does not carry over. + +For example, an application specifying 4.0 CPU has access to 400 ms of +CPU time every 100 ms. This CPU quota can be used in different ways, +depending on the application and available resources. Consider the +scenarios shown in this diagram. + + + +* *Scenario A*: the application can use up to 4 cores continuously for +every 100 ms interval. It is never throttled and starts processing +new requests immediately. + +* *Scenario B* : the application uses up to 8 cores (depending on +availability) but is throttled after 50 ms. The CPU quota resets at the +start of each new 100 ms interval. + +* *Scenario C* : is like Scenario A, but there is a garbage collection +event in the second interval that consumes all CPU quota. The +application throttles for the remaining 75 ms of that interval and +cannot service requests until the next interval. In this example, the +garbage collection finished in one interval but, depending on how much +garbage needs collecting, it may take more than one interval and further +delay service of requests. + +*Technical Note*: Mesos considers logical cores, also known as +hyperthreading or SMT cores, as the unit of CPU. + +## CPU Sizing + +To correctly size Aurora-run Mesos tasks, specify a per-shard CPU value +that lets the task run at its desired performance when at peak load +distributed across all shards. Include reserve capacity of at least 50%, +possibly more, depending on how critical your service is (or how +confident you are about your original estimate : -)), ideally by +increasing the number of shards to also improve resiliency. When running +your application, observe its CPU stats over time. If consistently at or +near your quota during peak load, you should consider increasing either +per-shard CPU or the number of shards. + +## Memory Isolation + +Mesos uses dedicated memory allocation. Your application always has +access to the amount of memory specified in your configuration. The +application's memory use is defined as the sum of the resident set size +(RSS) of all processes in a shard. Each shard is considered +independently. + +In other words, say you specified a memory size of 10GB. Each shard +would receive 10GB of memory. If an individual shard's memory demands +exceed 10GB, that shard is killed, but the other shards continue +working. + +*Technical note*: Total memory size is not enforced at allocation time, +so your application can request more than its allocation without getting +an ENOMEM. However, it will be killed shortly after. + +## Memory Sizing + +Size for your application's peak requirement. Observe the per-instance +memory statistics over time, as memory requirements can vary over +different periods. Remember that if your application exceeds its memory +value, it will be killed, so you should also add a safety margin of +around 10-20%. If you have the ability to do so, you may also want to +put alerts on the per-instance memory. + +## Disk Space + +Disk space used by your application is defined as the sum of the files' +disk space in your application's directory, including the `stdout` and +`stderr` logged from your application. Each shard is considered +independently. You should use off-node storage for your application's +data whenever possible. + +In other words, say you specified disk space size of 100MB. Each shard +would receive 100MB of disk space. If an individual shard's disk space +demands exceed 100MB, that shard is killed, but the other shards +continue working. + +After your application finishes running, its allocated disk space is +reclaimed. Thus, your job's final action should move any disk content +that you want to keep, such as logs, to your home file system or other +less transitory storage. Disk reclamation takes place an undefined +period after the application finish time; until then, the disk contents +are still available but you shouldn't count on them being so. + +*Technical note* : Disk space is not enforced at write so your +application can write above its quota without getting an ENOSPC, but it +will be killed shortly after. This is subject to change. + +## Disk Space Sizing + +Size for your application's peak requirement. Rotate and discard log +files as needed to stay within your quota. When running a Java process, +add the maximum size of the Java heap to your disk space requirement, in +order to account for an out of memory error dumping the heap +into the application's sandbox space. + +## Other Resources + +Other resources, such as network bandwidth, do not have any performance +guarantees. For some resources, such as memory bandwidth, there are no +practical sharing methods so some application combinations collocated on +the same host may cause contention. + +## Resource Quota + +Aurora requires resource quotas for +[production non-dedicated jobs](/documentation/0.12.0/configuration-reference/#job-objects). Quota is enforced at +the job role level and when set, defines a non-preemptible pool of compute resources within +that role. + +To grant quota to a particular role in production use `aurora_admin set_quota` command. + +NOTE: all job types (service, adhoc or cron) require role resource quota unless a job has +[dedicated constraint set](/documentation/0.12.0/deploying-aurora-scheduler/#dedicated-attribute). + +## Task preemption + +Under a particular resource shortage pressure, tasks from +[production](/documentation/0.12.0/configuration-reference/#job-objects) jobs may preempt tasks from any non-production +job. A production task may only be preempted by tasks from production jobs in the same role with +higher [priority](/documentation/0.12.0/configuration-reference/#job-objects). \ No newline at end of file Added: aurora/site/source/documentation/0.12.0/security.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/security.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/security.md (added) +++ aurora/site/source/documentation/0.12.0/security.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,279 @@ +Aurora integrates with [Apache Shiro](http://shiro.apache.org/) to provide security +controls for its API. In addition to providing some useful features out of the box, Shiro +also allows Aurora cluster administrators to adapt the security system to their organizationâs +existing infrastructure. + +- [Enabling Security](#enabling-security) +- [Authentication](#authentication) + - [HTTP Basic Authentication](#http-basic-authentication) + - [Server Configuration](#server-configuration) + - [Client Configuration](#client-configuration) + - [HTTP SPNEGO Authentication (Kerberos)](#http-spnego-authentication-kerberos) + - [Server Configuration](#server-configuration-1) + - [Client Configuration](#client-configuration-1) +- [Authorization](#authorization) + - [Using an INI file to define security controls](#using-an-ini-file-to-define-security-controls) + - [Caveats](#caveats) +- [Implementing a Custom Realm](#implementing-a-custom-realm) + - [Packaging a realm module](#packaging-a-realm-module) +- [Known Issues](#known-issues) + +# Enabling Security + +There are two major components of security: +[authentication and authorization](http://en.wikipedia.org/wiki/Authentication#Authorization). A +cluster administrator may choose the approach used for each, and may also implement custom +mechanisms for either. Later sections describe the options available. + +# Authentication + +The scheduler must be configured with instructions for how to process authentication +credentials at a minimum. There are currently two built-in authentication schemes - +[HTTP Basic Authentication](http://en.wikipedia.org/wiki/Basic_access_authentication), and +[SPNEGO](http://en.wikipedia.org/wiki/SPNEGO) (Kerberos). + +## HTTP Basic Authentication + +Basic Authentication is a very quick way to add *some* security. It is supported +by all major browsers and HTTP client libraries with minimal work. However, +before relying on Basic Authentication you should be aware of the [security +considerations](http://tools.ietf.org/html/rfc2617#section-4). + +### Server Configuration + +At a minimum you need to set 4 command-line flags on the scheduler: + +``` +-http_authentication_mechanism=BASIC +-shiro_realm_modules=INI_AUTHNZ +-shiro_ini_path=path/to/security.ini +``` + +And create a security.ini file like so: + +``` +[users] +sally = apple, admin + +[roles] +admin = * +``` + +The details of the security.ini file are explained below. Note that this file contains plaintext, +unhashed passwords. + +### Client Configuration + +To configure the client for HTTP Basic authentication, add an entry to ~/.netrc with your credentials + +``` +% cat ~/.netrc +# ... + +machine aurora.example.com +login sally +password apple + +# ... +``` + +No changes are required to `clusters.json`. + +## HTTP SPNEGO Authentication (Kerberos) + +### Server Configuration +At a minimum you need to set 6 command-line flags on the scheduler: + +``` +-http_authentication_mechanism=NEGOTIATE +-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ +-kerberos_server_principal=HTTP/[email protected] +-kerberos_server_keytab=path/to/aurora.example.com.keytab +-shiro_ini_path=path/to/security.ini +``` + +And create a security.ini file like so: + +``` +% cat path/to/security.ini +[users] +sally = _, admin + +[roles] +admin = * +``` + +What's going on here? First, Aurora must be configured to request Kerberos credentials when presented with an +unauthenticated request. This is achieved by setting + +``` +-http_authentication_mechanism=NEGOTIATE +``` + +Next, a Realm module must be configured to **authenticate** the current request using the Kerberos +credentials that were requested. Aurora ships with a realm module that can do this + +``` +-shiro_realm_modules=KERBEROS5_AUTHN[,...] +``` + +The Kerberos5Realm requires a keytab file and a server principal name. The principal name will usually +be in the form `HTTP/[email protected]`. + +``` +-kerberos_server_principal=HTTP/[email protected] +-kerberos_server_keytab=path/to/aurora.example.com.keytab +``` + +The Kerberos5 realm module is authentication-only. For scheduler security to work you must also +enable a realm module that provides an Authorizer implementation. For example, to do this using the +IniShiroRealmModule: + +``` +-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ +``` + +You can then configure authorization using a security.ini file as described below +(the password field is ignored). You must configure the realm module with the path to this file: + +``` +-shiro_ini_path=path/to/security.ini +``` + +### Client Configuration +To use Kerberos on the client-side you must build Kerberos-enabled client binaries. Do this with + +``` +./pants binary src/main/python/apache/aurora/kerberos:kaurora +./pants binary src/main/python/apache/aurora/kerberos:kaurora_admin +``` + +You must also configure each cluster where you've enabled Kerberos on the scheduler +to use Kerberos authentication. Do this by setting `auth_mechanism` to `KERBEROS` +in `clusters.json`. + +``` +% cat ~/.aurora/clusters.json +{ + "devcluser": { + "auth_mechanism": "KERBEROS", + ... + }, + ... +} +``` + +# Authorization +Given a means to authenticate the entity a client claims they are, we need to define what privileges they have. + +## Using an INI file to define security controls + +The simplest security configuration for Aurora is an INI file on the scheduler. For small +clusters, or clusters where the users and access controls change relatively infrequently, this is +likely the preferred approach. However you may want to avoid this approach if access permissions +are rapidly changing, or if your access control information already exists in another system. + +You can enable INI-based configuration with following scheduler command line arguments: + +``` +-http_authentication_mechanism=BASIC +-shiro_ini_path=path/to/security.ini +``` + +*note* As the argument name reveals, this is using Shiroâs +[IniRealm](http://shiro.apache.org/configuration.html#Configuration-INIConfiguration) behind +the scenes. + +The INI file will contain two sections - users and roles. Hereâs an example for what might +be in security.ini: + +``` +[users] +sally = apple, admin +jim = 123456, accounting +becky = letmein, webapp +larry = 654321,accounting +steve = password + +[roles] +admin = * +accounting = thrift.AuroraAdmin:setQuota +webapp = thrift.AuroraSchedulerManager:*:webapp +``` + +The users section defines user user credentials and the role(s) they are members of. These lines +are of the format `<user> = <password>[, <role>...]`. As you probably noticed, the passwords are +in plaintext and as a result read access to this file should be restricted. + +In this configuration, each user has different privileges for actions in the cluster because +of the roles they are a part of: + +* admin is granted all privileges +* accounting may adjust the amount of resource quota for any role +* webapp represents a collection of jobs that represents a service, and its members may create and modify any jobs owned by it + +### Caveats +You might find documentation on the Internet suggesting there are additional sections in `shiro.ini`, +like `[main]` and `[urls]`. These are not supported by Aurora as it uses a different mechanism to configure +those parts of Shiro. Think of Aurora's `security.ini` as a subset with only `[users]` and `[roles]` sections. + +## Implementing Delegated Authorization + +It is possible to leverage Shiro's `runAs` feature by implementing a custom Servlet Filter that provides +the capability and passing it's fully qualified class name to the command line argument +`-shiro_after_auth_filter`. The filter is registered in the same filter chain as the Shiro auth filters +and is placed after the Shiro auth filters in the filter chain. This ensures that the Filter is invoked +after the Shiro filters have had a chance to authenticate the request. + +# Implementing a Custom Realm + +Since Auroraâs security is backed by [Apache Shiro](https://shiro.apache.org), you can implement a +custom [Realm](http://shiro.apache.org/realm.html) to define organization-specific security behavior. + +In addition to using Shiro's standard APIs to implement a Realm you can link against Aurora to +access the type-safe Permissions Aurora uses. See the Javadoc for `org.apache.aurora.scheduler.spi` +for more information. + +## Packaging a realm module +Package your custom Realm(s) with a Guice module that exposes a `Set<Realm>` multibinding. + +```java +package com.example; + +import com.google.inject.AbstractModule; +import com.google.inject.multibindings.Multibinder; +import org.apache.shiro.realm.Realm; + +public class MyRealmModule extends AbstractModule { + @Override + public void configure() { + Realm myRealm = new MyRealm(); + + Multibinder.newSetBinder(binder(), Realm.class).addBinding().toInstance(myRealm); + } + + static class MyRealm implements Realm { + // Realm implementation. + } +} +``` + +To use your module in the scheduler, include it as a realm module based on its fully-qualified +class name: + +``` +-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ,com.example.MyRealmModule +``` + +# Known Issues + +While the APIs and SPIs we ship with are stable as of 0.8.0, we are aware of several incremental +improvements. Please follow, vote, or send patches. + +Relevant tickets: +* [AURORA-343](https://issues.apache.org/jira/browse/AURORA-343): HTTPS support +* [AURORA-1248](https://issues.apache.org/jira/browse/AURORA-1248): Client retries 4xx errors +* [AURORA-1279](https://issues.apache.org/jira/browse/AURORA-1279): Remove kerberos-specific build targets +* [AURORA-1293](https://issues.apache.org/jira/browse/AURORA-1291): Consider defining a JSON format in place of INI +* [AURORA-1179](https://issues.apache.org/jira/browse/AURORA-1179): Supported hashed passwords in security.ini +* [AURORA-1295](https://issues.apache.org/jira/browse/AURORA-1295): Support security for the ReadOnlyScheduler service Added: aurora/site/source/documentation/0.12.0/sla.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.12.0/sla.md?rev=1733548&view=auto ============================================================================== --- aurora/site/source/documentation/0.12.0/sla.md (added) +++ aurora/site/source/documentation/0.12.0/sla.md Fri Mar 4 02:43:01 2016 @@ -0,0 +1,177 @@ +Aurora SLA Measurement +-------------- + +- [Overview](#overview) +- [Metric Details](#metric-details) + - [Platform Uptime](#platform-uptime) + - [Job Uptime](#job-uptime) + - [Median Time To Assigned (MTTA)](#median-time-to-assigned-\(mtta\)) + - [Median Time To Running (MTTR)](#median-time-to-running-\(mttr\)) +- [Limitations](#limitations) + +## Overview + +The primary goal of the feature is collection and monitoring of Aurora job SLA (Service Level +Agreements) metrics that defining a contractual relationship between the Aurora/Mesos platform +and hosted services. + +The Aurora SLA feature is by default only enabled for service (non-cron) +production jobs (`"production = True"` in your `.aurora` config). It can be enabled for +non-production services via the scheduler command line flag `-sla_non_prod_metrics`. + +Counters that track SLA measurements are computed periodically within the scheduler. +The individual instance metrics are refreshed every minute (configurable via +`sla_stat_refresh_interval`). The instance counters are subsequently aggregated by +relevant grouping types before exporting to scheduler `/vars` endpoint (when using `vagrant` +that would be `http://192.168.33.7:8081/vars`) + +## Metric Details + +### Platform Uptime + +*Aggregate amount of time a job spends in a non-runnable state due to platform unavailability +or scheduling delays. This metric tracks Aurora/Mesos uptime performance and reflects on any +system-caused downtime events (tasks LOST or DRAINED). Any user-initiated task kills/restarts +will not degrade this metric.* + +**Collection scope:** + +* Per job - `sla_<job_key>_platform_uptime_percent` +* Per cluster - `sla_cluster_platform_uptime_percent` + +**Units:** percent + +A fault in the task environment may cause the Aurora/Mesos to have different views on the task state +or lose track of the task existence. In such cases, the service task is marked as LOST and +rescheduled by Aurora. For example, this may happen when the task stays in ASSIGNED or STARTING +for too long or the Mesos slave becomes unhealthy (or disappears completely). The time between +task entering LOST and its replacement reaching RUNNING state is counted towards platform downtime. + +Another example of a platform downtime event is the administrator-requested task rescheduling. This +happens during planned Mesos slave maintenance when all slave tasks are marked as DRAINED and +rescheduled elsewhere. + +To accurately calculate Platform Uptime, we must separate platform incurred downtime from user +actions that put a service instance in a non-operational state. It is simpler to isolate +user-incurred downtime and treat all other downtime as platform incurred. + +Currently, a user can cause a healthy service (task) downtime in only two ways: via `killTasks` +or `restartShards` RPCs. For both, their affected tasks leave an audit state transition trail +relevant to uptime calculations. By applying a special "SLA meaning" to exposed task state +transition records, we can build a deterministic downtime trace for every given service instance. + +A task going through a state transition carries one of three possible SLA meanings +(see [SlaAlgorithm.java](https://github.com/apache/aurora/blob/#{git_tag}/src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java)) for +sla-to-task-state mapping): + +* Task is UP: starts a period where the task is considered to be up and running from the Aurora + platform standpoint. + +* Task is DOWN: starts a period where the task cannot reach the UP state for some + non-user-related reason. Counts towards instance downtime. + +* Task is REMOVED from SLA: starts a period where the task is not expected to be UP due to + user initiated action or failure. We ignore this period for the uptime calculation purposes. + +This metric is recalculated over the last sampling period (last minute) to account for +any UP/DOWN/REMOVED events. It ignores any UP/DOWN events not immediately adjacent to the +sampling interval as well as adjacent REMOVED events. + +### Job Uptime + +*Percentage of the job instances considered to be in RUNNING state for the specified duration +relative to request time. This is a purely application side metric that is considering aggregate +uptime of all RUNNING instances. Any user- or platform initiated restarts directly affect +this metric.* + +**Collection scope:** We currently expose job uptime values at 5 pre-defined +percentiles (50th,75th,90th,95th and 99th): + +* `sla_<job_key>_job_uptime_50_00_sec` +* `sla_<job_key>_job_uptime_75_00_sec` +* `sla_<job_key>_job_uptime_90_00_sec` +* `sla_<job_key>_job_uptime_95_00_sec` +* `sla_<job_key>_job_uptime_99_00_sec` + +**Units:** seconds +You can also get customized real-time stats from aurora client. See `aurora sla -h` for +more details. + +### Median Time To Assigned (MTTA) + +*Median time a job spends waiting for its tasks to be assigned to a host. This is a combined +metric that helps track the dependency of scheduling performance on the requested resources +(user scope) as well as the internal scheduler bin-packing algorithm efficiency (platform scope).* + +**Collection scope:** + +* Per job - `sla_<job_key>_mtta_ms` +* Per cluster - `sla_cluster_mtta_ms` +* Per instance size (small, medium, large, x-large, xx-large). Size are defined in: +[ResourceAggregates.java](https://github.com/apache/aurora/blob/#{git_tag}/src/main/java/org/apache/aurora/scheduler/base/ResourceAggregates.java)) + * By CPU: + * `sla_cpu_small_mtta_ms` + * `sla_cpu_medium_mtta_ms` + * `sla_cpu_large_mtta_ms` + * `sla_cpu_xlarge_mtta_ms` + * `sla_cpu_xxlarge_mtta_ms` + * By RAM: + * `sla_ram_small_mtta_ms` + * `sla_ram_medium_mtta_ms` + * `sla_ram_large_mtta_ms` + * `sla_ram_xlarge_mtta_ms` + * `sla_ram_xxlarge_mtta_ms` + * By DISK: + * `sla_disk_small_mtta_ms` + * `sla_disk_medium_mtta_ms` + * `sla_disk_large_mtta_ms` + * `sla_disk_xlarge_mtta_ms` + * `sla_disk_xxlarge_mtta_ms` + +**Units:** milliseconds + +MTTA only considers instances that have already reached ASSIGNED state and ignores those +that are still PENDING. This ensures straggler instances (e.g. with unreasonable resource +constraints) do not affect metric curves. + +### Median Time To Running (MTTR) + +*Median time a job waits for its tasks to reach RUNNING state. This is a comprehensive metric +reflecting on the overall time it takes for the Aurora/Mesos to start executing user content.* + +**Collection scope:** + +* Per job - `sla_<job_key>_mttr_ms` +* Per cluster - `sla_cluster_mttr_ms` +* Per instance size (small, medium, large, x-large, xx-large). Size are defined in: +[ResourceAggregates.java](https://github.com/apache/aurora/blob/#{git_tag}/src/main/java/org/apache/aurora/scheduler/base/ResourceAggregates.java)) + * By CPU: + * `sla_cpu_small_mttr_ms` + * `sla_cpu_medium_mttr_ms` + * `sla_cpu_large_mttr_ms` + * `sla_cpu_xlarge_mttr_ms` + * `sla_cpu_xxlarge_mttr_ms` + * By RAM: + * `sla_ram_small_mttr_ms` + * `sla_ram_medium_mttr_ms` + * `sla_ram_large_mttr_ms` + * `sla_ram_xlarge_mttr_ms` + * `sla_ram_xxlarge_mttr_ms` + * By DISK: + * `sla_disk_small_mttr_ms` + * `sla_disk_medium_mttr_ms` + * `sla_disk_large_mttr_ms` + * `sla_disk_xlarge_mttr_ms` + * `sla_disk_xxlarge_mttr_ms` + +**Units:** milliseconds + +MTTR only considers instances in RUNNING state. This ensures straggler instances (e.g. with +unreasonable resource constraints) do not affect metric curves. + +## Limitations + +* The availability of Aurora SLA metrics is bound by the scheduler availability. + +* All metrics are calculated at a pre-defined interval (currently set at 1 minute). + Scheduler restarts may result in missed collections.
