Added: aurora/site/publish/documentation/0.21.0/reference/scheduler-configuration/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.21.0/reference/scheduler-configuration/index.html?rev=1840515&view=auto ============================================================================== --- aurora/site/publish/documentation/0.21.0/reference/scheduler-configuration/index.html (added) +++ aurora/site/publish/documentation/0.21.0/reference/scheduler-configuration/index.html Tue Sep 11 05:28:10 2018 @@ -0,0 +1,424 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/reference/scheduler-configuration/'" + value="0.21.0"> + <option value="0.21.0" + selected="selected"> + 0.21.0 + (latest) + </option> + <option value="0.20.0" + > + 0.20.0 + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="scheduler-configuration-reference">Scheduler Configuration Reference</h1> + +<p>The Aurora scheduler can take a variety of configuration options through command-line arguments. +A list of the available options can be seen by running <code>aurora-scheduler -help</code>.</p> + +<p>Please refer to the <a href="../../operations/configuration/">Operator Configuration Guide</a> for details on how +to properly set the most important options.</p> +<pre class="highlight plaintext"><code>$ aurora-scheduler -help +------------------------------------------------------------------------- +-h or -help to print this help message + +Required flags: +-backup_dir [not null] + Directory to store backups under. Will be created if it does not exist. +-cluster_name [not null] + Name to identify the cluster being served. +-framework_authentication_file + Properties file which contains framework credentials to authenticate with Mesosmaster. Must contain the properties 'aurora_authentication_principal' and 'aurora_authentication_secret'. +-ip + The ip address to listen. If not set, the scheduler will listen on all interfaces. +-mesos_master_address [not null] + Address for the mesos master, can be a socket address or zookeeper path. +-mesos_role + The Mesos role this framework will register as. The default is to left this empty, and the framework will register without any role and only receive unreserved resources in offer. +-serverset_path [not null, must be non-empty] + ZooKeeper ServerSet path to register at. +-shiro_after_auth_filter + Fully qualified class name of the servlet filter to be applied after the shiro auth filters are applied. +-thermos_executor_path + Path to the thermos executor entry point. +-tier_config [file must be readable] + Configuration file defining supported task tiers, task traits and behaviors. +-webhook_config [file must exist, file must be readable] + Path to webhook configuration file. +-zk_endpoints [must have at least 1 item] + Endpoint specification for the ZooKeeper servers. + +Optional flags: +-allow_container_volumes (default false) + Allow passing in volumes in the job. Enabling this could pose a privilege escalation threat. +-allow_docker_parameters (default false) + Allow to pass docker container parameters in the job. +-allow_gpu_resource (default false) + Allow jobs to request Mesos GPU resource. +-allowed_container_types (default [MESOS]) + Container types that are allowed to be used by jobs. +-allowed_job_environments (default ^(prod|devel|test|staging\d*)$) + Regular expression describing the environments that are allowed to be used by jobs. +-async_slot_stat_update_interval (default (1, mins)) + Interval on which to try to update open slot stats. +-async_task_stat_update_interval (default (1, hrs)) + Interval on which to try to update resource consumption stats. +-async_worker_threads (default 8) + The number of worker threads to process async task operations with. +-backup_interval (default (1, hrs)) + Minimum interval on which to write a storage backup. +-cron_scheduler_num_threads (default 10) + Number of threads to use for the cron scheduler thread pool. +-cron_scheduling_max_batch_size (default 10) [must be > 0] + The maximum number of triggered cron jobs that can be processed in a batch. +-cron_start_initial_backoff (default (5, secs)) + Initial backoff delay while waiting for a previous cron run to be killed. +-cron_start_max_backoff (default (1, mins)) + Max backoff delay while waiting for a previous cron run to be killed. +-cron_timezone (default GMT) + TimeZone to use for cron predictions. +-custom_executor_config [file must exist, file must be readable] + Path to custom executor settings configuration file. +-default_docker_parameters (default {}) + Default docker parameters for any job that does not explicitly declare parameters. +-dlog_max_entry_size (default (512, KB)) + Specifies the maximum entry size to append to the log. Larger entries will be split across entry Frames. +-dlog_shutdown_grace_period (default (2, secs)) + Specifies the maximum time to wait for scheduled checkpoint and snapshot actions to complete before forcibly shutting down. +-dlog_snapshot_interval (default (1, hrs)) + Specifies the frequency at which snapshots of local storage are taken and written to the log. +-enable_cors_for + List of domains for which CORS support should be enabled. +-enable_mesos_fetcher (default false) + Allow jobs to pass URIs to the Mesos Fetcher. Note that enabling this feature could pose a privilege escalation threat. +-enable_preemptor (default true) + Enable the preemptor and preemption +-enable_revocable_cpus (default true) + Treat CPUs as a revocable resource. +-enable_revocable_ram (default false) + Treat RAM as a revocable resource. +-executor_user (default root) + User to start the executor. Defaults to "root". Set this to an unprivileged user if the mesos master was started with "--no-root_submissions". If set to anything other than "root", the executor will ignore the "role" setting for jobs since it can't use setuid() anymore. This means that all your jobs will run under the specified user and the user has to exist on the Mesos agents. +-first_schedule_delay (default (1, ms)) + Initial amount of time to wait before first attempting to schedule a PENDING task. +-flapping_task_threshold (default (5, mins)) + A task that repeatedly runs for less than this time is considered to be flapping. +-framework_announce_principal (default false) + When 'framework_authentication_file' flag is set, the FrameworkInfo registered with the mesos master will also contain the principal. This is necessary if you intend to use mesos authorization via mesos ACLs. The default will change in a future release. Changing this value is backwards incompatible. For details, see MESOS-703. +-framework_failover_timeout (default (21, days)) + Time after which a framework is considered deleted. SHOULD BE VERY HIGH. +-framework_name (default Aurora) + Name used to register the Aurora framework with Mesos. +-global_container_mounts (default []) + A comma separated list of mount points (in host:container form) to mount into all (non-mesos) containers. +-history_max_per_job_threshold (default 100) + Maximum number of terminated tasks to retain in a job history. +-history_min_retention_threshold (default (1, hrs)) + Minimum guaranteed time for task history retention before any pruning is attempted. +-history_prune_threshold (default (2, days)) + Time after which the scheduler will prune terminated task history. +-host_maintenance_polling_interval (default (1, minute)) + Interval between polling for pending host maintenance requests. +-hostname + The hostname to advertise in ZooKeeper instead of the locally-resolved hostname. +-http_authentication_mechanism (default NONE) + HTTP Authentication mechanism to use. +-http_port (default 0) + The port to start an HTTP server on. Default value will choose a random port. +-initial_flapping_task_delay (default (30, secs)) + Initial amount of time to wait before attempting to schedule a flapping task. +-initial_schedule_penalty (default (1, secs)) + Initial amount of time to wait before attempting to schedule a task that has failed to schedule. +-initial_task_kill_retry_interval (default (5, secs)) + When killing a task, retry after this delay if mesos has not responded, backing off up to transient_task_state_timeout +-job_update_history_per_job_threshold (default 10) + Maximum number of completed job updates to retain in a job update history. +-job_update_history_pruning_interval (default (15, mins)) + Job update history pruning interval. +-job_update_history_pruning_threshold (default (30, days)) + Time after which the scheduler will prune completed job update history. +-kerberos_debug (default false) + Produce additional Kerberos debugging output. +-kerberos_server_keytab + Path to the server keytab. +-kerberos_server_principal + Kerberos server principal to use, usually of the form HTTP/[email protected] +-max_flapping_task_delay (default (5, mins)) + Maximum delay between attempts to schedule a flapping task. +-max_leading_duration (default (1, days)) + After leading for this duration, the scheduler should commit suicide. +-max_parallel_coordinated_maintenance (default 10) + Maximum number of coordinators that can be contacted in parallel. +-max_registration_delay (default (1, mins)) + Max allowable delay to allow the driver to register before aborting +-max_reschedule_task_delay_on_startup (default (30, secs)) + Upper bound of random delay for pending task rescheduling on scheduler startup. +-max_saved_backups (default 48) + Maximum number of backups to retain before deleting the oldest backups. +-max_schedule_attempts_per_sec (default 40.0) + Maximum number of scheduling attempts to make per second. +-max_schedule_penalty (default (1, mins)) + Maximum delay between attempts to schedule a PENDING tasks. +-max_sla_duration_secs (default (2, hrs)) + Maximum duration window for which SLA requirements are to be satisfied. This does not apply to jobs that have a CoordinatorSlaPolicy. +-max_status_update_batch_size (default 1000) [must be > 0] + The maximum number of status updates that can be processed in a batch. +-max_task_event_batch_size (default 300) [must be > 0] + The maximum number of task state change events that can be processed in a batch. +-max_tasks_per_job (default 4000) [must be > 0] + Maximum number of allowed tasks in a single job. +-max_tasks_per_schedule_attempt (default 5) [must be > 0] + The maximum number of tasks to pick in a single scheduling attempt. +-max_update_instance_failures (default 20000) [must be > 0] + Upper limit on the number of failures allowed during a job update. This helps cap potentially unbounded entries into storage. +-min_offer_hold_time (default (5, mins)) + Minimum amount of time to hold a resource offer before declining. +-min_required_instances_for_sla_check (default 20) + Minimum number of instances required for a job to be eligible for SLA check. This does not apply to jobs that have a CoordinatorSlaPolicy. +-native_log_election_retries (default 20) + The maximum number of attempts to obtain a new log writer. +-native_log_election_timeout (default (15, secs)) + The timeout for a single attempt to obtain a new log writer. +-native_log_file_path + Path to a file to store the native log data in. If the parent directory doesnot exist it will be created. +-native_log_quorum_size (default 1) + The size of the quorum required for all log mutations. +-native_log_read_timeout (default (5, secs)) + The timeout for doing log reads. +-native_log_write_timeout (default (3, secs)) + The timeout for doing log appends and truncations. +-native_log_zk_group_path + A zookeeper node for use by the native log to track the master coordinator. +-offer_filter_duration (default (5, secs)) + Duration after which we expect Mesos to re-offer unused resources. A short duration improves scheduling performance in smaller clusters, but might lead to resource starvation for other frameworks if you run many frameworks in your cluster. +-offer_hold_jitter_window (default (1, mins)) + Maximum amount of random jitter to add to the offer hold time window. +-offer_reservation_duration (default (3, mins)) + Time to reserve a agent's offers while trying to satisfy a task preempting another. +-offer_set_module (default [class org.apache.aurora.scheduler.offers.OfferSetModule]) + Guice module for replacing offer holding and scheduling logic. +-partition_aware (default false) + Whether or not to integrate with the partition-aware Mesos capabilities. +-populate_discovery_info (default false) + If true, Aurora populates DiscoveryInfo field of Mesos TaskInfo. +-preemption_delay (default (3, mins)) + Time interval after which a pending task becomes eligible to preempt other tasks +-preemption_slot_finder_modules (default [class org.apache.aurora.scheduler.preemptor.PendingTaskProcessorModule, class org.apache.aurora.scheduler.preemptor.PreemptionVictimFilterModule]) + Guice modules for replacing preemption logic. +-preemption_slot_hold_time (default (5, mins)) + Time to hold a preemption slot found before it is discarded. +-preemption_slot_search_interval (default (1, mins)) + Time interval between pending task preemption slot searches. +-receive_revocable_resources (default false) + Allows receiving revocable resource offers from Mesos. +-reconciliation_explicit_batch_interval (default (5, secs)) + Interval between explicit batch reconciliation requests. +-reconciliation_explicit_batch_size (default 1000) [must be > 0] + Number of tasks in a single batch request sent to Mesos for explicit reconciliation. +-reconciliation_explicit_interval (default (60, mins)) + Interval on which scheduler will ask Mesos for status updates of all non-terminal tasks known to scheduler. +-reconciliation_implicit_interval (default (60, mins)) + Interval on which scheduler will ask Mesos for status updates of all non-terminal tasks known to Mesos. +-reconciliation_initial_delay (default (1, mins)) + Initial amount of time to delay task reconciliation after scheduler start up. +-reconciliation_schedule_spread (default (30, mins)) + Difference between explicit and implicit reconciliation intervals intended to create a non-overlapping task reconciliation schedule. +-require_docker_use_executor (default true) + If false, Docker tasks may run without an executor (EXPERIMENTAL) +-scheduling_max_batch_size (default 3) [must be > 0] + The maximum number of scheduling attempts that can be processed in a batch. +-serverset_endpoint_name (default http) + Name of the scheduler endpoint published in ZooKeeper. +-shiro_ini_path + Path to shiro.ini for authentication and authorization configuration. +-shiro_realm_modules (default [class org.apache.aurora.scheduler.http.api.security.IniShiroRealmModule]) + Guice modules for configuring Shiro Realms. +-sla_aware_action_max_batch_size (default 300) [must be > 0] + The maximum number of sla aware update actions that can be processed in a batch. +-sla_aware_kill_retry_min_delay (default (1, min)) [must be > 0] + The minimum amount of time to wait before retrying an SLA-aware kill (using a truncated binary backoff). +-sla_aware_kill_retry_max_delay (default (5, min)) [must be > 0] + The maximum amount of time to wait before retrying an SLA-aware kill (using a truncated binary backoff). +-sla_coordinator_timeout (default (1, min)) [must be > 0] + Timeout interval for communicating with Coordinator. +-sla_non_prod_metrics (default []) + Metric categories collected for non production tasks. +-sla_prod_metrics (default [JOB_UPTIMES, PLATFORM_UPTIME, MEDIANS]) + Metric categories collected for production tasks. +-sla_stat_refresh_interval (default (1, mins)) + The SLA stat refresh interval. +-stat_retention_period (default (1, hrs)) + Time for a stat to be retained in memory before expiring. +-stat_sampling_interval (default (1, secs)) + Statistic value sampling interval. +-task_assigner_modules (default [class org.apache.aurora.scheduler.scheduling.TaskAssignerImplModule]) + Guice modules for replacing task assignment logic. +-thermos_executor_cpu (default 0.25) + The number of CPU cores to allocate for each instance of the executor. +-thermos_executor_flags + Extra arguments to be passed to the thermos executor +-thermos_executor_ram (default (128, MB)) + The amount of RAM to allocate for each instance of the executor. +-thermos_executor_resources (default []) + A comma separated list of additional resources to copy into the sandbox.Note: if thermos_executor_path is not the thermos_executor.pex file itself, this must include it. +-thermos_home_in_sandbox (default false) + If true, changes HOME to the sandbox before running the executor. This primarily has the effect of causing the executor and runner to extract themselves into the sandbox. +-thrift_method_interceptor_modules (default []) + Additional Guice modules for intercepting Thrift method calls. +-transient_task_state_timeout (default (5, mins)) + The amount of time after which to treat a task stuck in a transient state as LOST. +-viz_job_url_prefix (default ) + URL prefix for job container stats. +-zk_chroot_path + chroot path to use for the ZooKeeper connections +-zk_digest_credentials + user:password to use when authenticating with ZooKeeper. +-zk_in_proc (default false) + Launches an embedded zookeeper server for local testing causing -zk_endpoints to be ignored if specified. +-zk_session_timeout (default (4, secs)) + The ZooKeeper session timeout. +-zk_use_curator (default true) + DEPRECATED: Uses Apache Curator as the zookeeper client; otherwise a copy of Twitter commons/zookeeper (the legacy library) is used. +------------------------------------------------------------------------- +</code></pre> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html>
Added: aurora/site/publish/documentation/0.21.0/reference/scheduler-endpoints/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.21.0/reference/scheduler-endpoints/index.html?rev=1840515&view=auto ============================================================================== --- aurora/site/publish/documentation/0.21.0/reference/scheduler-endpoints/index.html (added) +++ aurora/site/publish/documentation/0.21.0/reference/scheduler-endpoints/index.html Tue Sep 11 05:28:10 2018 @@ -0,0 +1,176 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/reference/scheduler-endpoints/'" + value="0.21.0"> + <option value="0.21.0" + selected="selected"> + 0.21.0 + (latest) + </option> + <option value="0.20.0" + > + 0.20.0 + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="http-endpoints">HTTP endpoints</h1> + +<p>There are a number of HTTP endpoints that the Aurora scheduler exposes. These allow various +operational tasks to be performed on the scheduler. Below is an (incomplete) list of such endpoints +and a brief explanation of what they do.</p> + +<h2 id="leader-health">Leader health</h2> + +<p>The /leaderhealth endpoint enables performing health checks on the scheduler instances inorder +to forward requests to the leading scheduler. This is typically used by a load balancer such as +HAProxy or AWS ELB.</p> + +<p>When a HTTP GET request is issued on this endpoint, it responds as follows:</p> + +<ul> +<li>If the instance that received the GET request is the leading scheduler, a HTTP status code of +<code>200 OK</code> is returned.</li> +<li>If the instance that received the GET request is not the leading scheduler but a leader does +exist, a HTTP status code of <code>503 SERVICE_UNAVAILABLE</code> is returned.</li> +<li>If no leader currently exists or the leader is unknown, a HTTP status code of <code>502 BAD_GATEWAY</code> +is returned.</li> +</ul> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/0.21.0/reference/task-lifecycle/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.21.0/reference/task-lifecycle/index.html?rev=1840515&view=auto ============================================================================== --- aurora/site/publish/documentation/0.21.0/reference/task-lifecycle/index.html (added) +++ aurora/site/publish/documentation/0.21.0/reference/task-lifecycle/index.html Tue Sep 11 05:28:10 2018 @@ -0,0 +1,328 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/reference/task-lifecycle/'" + value="0.21.0"> + <option value="0.21.0" + selected="selected"> + 0.21.0 + (latest) + </option> + <option value="0.20.0" + > + 0.20.0 + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="task-lifecycle">Task Lifecycle</h1> + +<p>When Aurora reads a configuration file and finds a <code>Job</code> definition, it:</p> + +<ol> +<li> Evaluates the <code>Job</code> definition.</li> +<li> Splits the <code>Job</code> into its constituent <code>Task</code>s.</li> +<li> Sends those <code>Task</code>s to the scheduler.</li> +<li> The scheduler puts the <code>Task</code>s into <code>PENDING</code> state, starting each +<code>Task</code>’s life cycle.</li> +</ol> + +<p><img alt="Life of a task" src="../../images/lifeofatask.png" /></p> + +<p>Please note, a couple of task states described below are missing from +this state diagram.</p> + +<h2 id="pending-to-running-states">PENDING to RUNNING states</h2> + +<p>When a <code>Task</code> is in the <code>PENDING</code> state, the scheduler constantly +searches for machines satisfying that <code>Task</code>’s resource request +requirements (RAM, disk space, CPU time) while maintaining configuration +constraints such as “a <code>Task</code> must run on machines dedicated to a +particular role” or attribute limit constraints such as “at most 2 +<code>Task</code>s from the same <code>Job</code> may run on each rack”. When the scheduler +finds a suitable match, it assigns the <code>Task</code> to a machine and puts the +<code>Task</code> into the <code>ASSIGNED</code> state.</p> + +<p>From the <code>ASSIGNED</code> state, the scheduler sends an RPC to the agent +machine containing <code>Task</code> configuration, which the agent uses to spawn +an executor responsible for the <code>Task</code>’s lifecycle. When the scheduler +receives an acknowledgment that the machine has accepted the <code>Task</code>, +the <code>Task</code> goes into <code>STARTING</code> state.</p> + +<p><code>STARTING</code> state initializes a <code>Task</code> sandbox. When the sandbox is fully +initialized, Thermos begins to invoke <code>Process</code>es. Also, the agent +machine sends an update to the scheduler that the <code>Task</code> is +in <code>RUNNING</code> state, only after the task satisfies the liveness requirements. +See <a href="../features/services#health-checking">Health Checking</a> for more details +for how to configure health checks.</p> + +<h2 id="running-to-terminal-states">RUNNING to terminal states</h2> + +<p>There are various ways that an active <code>Task</code> can transition into a terminal +state. By definition, it can never leave this state. However, depending on +nature of the termination and the originating <code>Job</code> definition +(e.g. <code>service</code>, <code>max_task_failures</code>), a replacement <code>Task</code> might be +scheduled.</p> + +<h3 id="natural-termination-finished-failed">Natural Termination: FINISHED, FAILED</h3> + +<p>A <code>RUNNING</code> <code>Task</code> can terminate without direct user interaction. For +example, it may be a finite computation that finishes, even something as +simple as <code>echo hello world.</code>, or it could be an exceptional condition in +a long-lived service. If the <code>Task</code> is successful (its underlying +processes have succeeded with exit status <code>0</code> or finished without +reaching failure limits) it moves into <code>FINISHED</code> state. If it finished +after reaching a set of failure limits, it goes into <code>FAILED</code> state.</p> + +<p>A terminated <code>TASK</code> which is subject to rescheduling will be temporarily +<code>THROTTLED</code>, if it is considered to be flapping. A task is flapping, if its +previous invocation was terminated after less than 5 minutes (scheduler +default). The time penalty a task has to remain in the <code>THROTTLED</code> state, +before it is eligible for rescheduling, increases with each consecutive +failure.</p> + +<h3 id="forceful-termination-killing-restarting">Forceful Termination: KILLING, RESTARTING</h3> + +<p>You can terminate a <code>Task</code> by issuing an <code>aurora job kill</code> command, which +moves it into <code>KILLING</code> state. The scheduler then sends the agent a +request to terminate the <code>Task</code>. If the scheduler receives a successful +response, it moves the Task into <code>KILLED</code> state and never restarts it.</p> + +<p>If a <code>Task</code> is forced into the <code>RESTARTING</code> state via the <code>aurora job restart</code> +command, the scheduler kills the underlying task but in parallel schedules +an identical replacement for it.</p> + +<p>In any case, the responsible executor on the agent follows an escalation +sequence when killing a running task:</p> + +<ol> +<li>If a <code>HttpLifecycleConfig</code> is not present, skip to (4).</li> +<li>Send a POST to the <code>graceful_shutdown_endpoint</code> and wait +<code>graceful_shutdown_wait_secs</code> seconds.</li> +<li>Send a POST to the <code>shutdown_endpoint</code> and wait +<code>shutdown_wait_secs</code> seconds.</li> +<li>Send SIGTERM (<code>kill</code>) and wait at most <code>finalization_wait</code> seconds.</li> +<li>Send SIGKILL (<code>kill -9</code>).</li> +</ol> + +<p>If the executor notices that all <code>Process</code>es in a <code>Task</code> have aborted +during this sequence, it will not proceed with subsequent steps. +Note that graceful shutdown is best-effort, and due to the many +inevitable realities of distributed systems, it may not be performed.</p> + +<h3 id="unexpected-termination-lost">Unexpected Termination: LOST</h3> + +<p>If a <code>Task</code> stays in a transient task state for too long (such as <code>ASSIGNED</code> +or <code>STARTING</code>), the scheduler forces it into <code>LOST</code> state, creating a new +<code>Task</code> in its place that’s sent into <code>PENDING</code> state.</p> + +<p>In addition, if the Mesos core tells the scheduler that a agent has +become unhealthy (or outright disappeared), the <code>Task</code>s assigned to that +agent go into <code>LOST</code> state and new <code>Task</code>s are created in their place. +From <code>PENDING</code> state, there is no guarantee a <code>Task</code> will be reassigned +to the same machine unless job constraints explicitly force it there.</p> + +<h2 id="running-to-partitioned-states">RUNNING to PARTITIONED states</h2> + +<p>If Aurora is configured to enable partition awareness, a task which is in a +running state can transition to <code>PARTITIONED</code>. This happens when the state +of the task in Mesos becomes unknown. By default Aurora errs on the side of +availability, so all tasks that transition to <code>PARTITIONED</code> are immediately +transitioned to <code>LOST</code>.</p> + +<p>This policy is not ideal for all types of workloads you may wish to run in +your Aurora cluster, e.g. for jobs where task failures are very expensive. +So job owners may set their own <code>PartitionPolicy</code> where they can control +how long to remain in <code>PARTITIONED</code> before transitioning to <code>LOST</code>. Or they +can disable any automatic attempts to <code>reschedule</code> when in <code>PARTITIONED</code>, +effectively waiting out the partition for as long as possible.</p> + +<h2 id="partitioned-and-transient-states">PARTITIONED and transient states</h2> + +<p>The <code>PartitionPolicy</code> provided by users only applies to tasks which are +currently running. When tasks are moving in and out of transient states, +e.g. tasks being updated, restarted, preempted, etc., <code>PARTITIONED</code> tasks +are moved immediately to <code>LOST</code>. This prevents situations where system +or user-initiated actions are blocked indefinitely waiting for partitions +to resolve (that may never be resolved).</p> + +<h3 id="giving-priority-to-production-tasks-preempting">Giving Priority to Production Tasks: PREEMPTING</h3> + +<p>Sometimes a Task needs to be interrupted, such as when a non-production +Task’s resources are needed by a higher priority production Task. This +type of interruption is called a <em>pre-emption</em>. When this happens in +Aurora, the non-production Task is killed and moved into +the <code>PREEMPTING</code> state when both the following are true:</p> + +<ul> +<li>The task being killed is a non-production task.</li> +<li>The other task is a <code>PENDING</code> production task that hasn’t been +scheduled due to a lack of resources.</li> +</ul> + +<p>The scheduler UI shows the non-production task was preempted in favor of +the production task. At some point, tasks in <code>PREEMPTING</code> move to <code>KILLED</code>.</p> + +<p>Note that non-production tasks consuming many resources are likely to be +preempted in favor of production tasks.</p> + +<h3 id="making-room-for-maintenance-draining">Making Room for Maintenance: DRAINING</h3> + +<p>Cluster operators can set agent into maintenance mode. This will transition +all <code>Task</code> running on this agent into <code>DRAINING</code> and eventually to <code>KILLED</code>. +Drained <code>Task</code>s will be restarted on other agents for which no maintenance +has been announced yet.</p> + +<h2 id="state-reconciliation">State Reconciliation</h2> + +<p>Due to the many inevitable realities of distributed systems, there might +be a mismatch of perceived and actual cluster state (e.g. a machine returns +from a <code>netsplit</code> but the scheduler has already marked all its <code>Task</code>s as +<code>LOST</code> and rescheduled them).</p> + +<p>Aurora regularly runs a state reconciliation process in order to detect +and correct such issues (e.g. by killing the errant <code>RUNNING</code> tasks). +By default, the proper detection of all failure scenarios and inconsistencies +may take up to an hour.</p> + +<p>To emphasize this point: there is no uniqueness guarantee for a single +instance of a job in the presence of network partitions. If the <code>Task</code> +requires that, it should be baked in at the application level using a +distributed coordination service such as Zookeeper.</p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/publish/documentation/latest/features/sla-requirements/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/features/sla-requirements/index.html?rev=1840515&view=auto ============================================================================== --- aurora/site/publish/documentation/latest/features/sla-requirements/index.html (added) +++ aurora/site/publish/documentation/latest/features/sla-requirements/index.html Tue Sep 11 05:28:10 2018 @@ -0,0 +1,334 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Apache Aurora</title> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css";> + <link href="/assets/css/main.css" rel="stylesheet"> + <!-- Analytics --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-45879646-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + </head> + <body> + <div class="container-fluid section-header"> + <div class="container"> + <div class="nav nav-bar"> + <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/community/">Community</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/blog/">Blog</a></li> + </ul> + </div> + </div> +</div> + + <div class="container-fluid"> + <div class="container content"> + <div class="col-md-12 documentation"> +<h5 class="page-header text-uppercase">Documentation +<select onChange="window.location.href='/documentation/' + this.value + '/features/sla-requirements/'" + value="latest"> + <option value="0.21.0" + > + 0.21.0 + (latest) + </option> + <option value="0.20.0" + > + 0.20.0 + </option> + <option value="0.19.1" + > + 0.19.1 + </option> + <option value="0.19.0" + > + 0.19.0 + </option> + <option value="0.18.1" + > + 0.18.1 + </option> + <option value="0.18.0" + > + 0.18.0 + </option> + <option value="0.17.0" + > + 0.17.0 + </option> + <option value="0.16.0" + > + 0.16.0 + </option> + <option value="0.15.0" + > + 0.15.0 + </option> + <option value="0.14.0" + > + 0.14.0 + </option> + <option value="0.13.0" + > + 0.13.0 + </option> + <option value="0.12.0" + > + 0.12.0 + </option> + <option value="0.11.0" + > + 0.11.0 + </option> + <option value="0.10.0" + > + 0.10.0 + </option> + <option value="0.9.0" + > + 0.9.0 + </option> + <option value="0.8.0" + > + 0.8.0 + </option> + <option value="0.7.0-incubating" + > + 0.7.0-incubating + </option> + <option value="0.6.0-incubating" + > + 0.6.0-incubating + </option> + <option value="0.5.0-incubating" + > + 0.5.0-incubating + </option> +</select> +</h5> +<h1 id="sla-requirements">SLA Requirements</h1> + +<ul> +<li><a href="#overview">Overview</a></li> +<li><a href="#default-sla">Default SLA</a></li> +<li><a href="#custom-sla">Custom SLA</a> + +<ul> +<li><a href="#count-based">Count-based</a></li> +<li><a href="#percentage-based">Percentage-based</a></li> +<li><a href="#coordinator-based">Coordinator-based</a></li> +</ul></li> +</ul> + +<h2 id="overview">Overview</h2> + +<p>Aurora guarantees SLA requirements for jobs. These requirements limit the impact of cluster-wide +maintenance operations on the jobs. For instance, when an operator upgrades +the OS on all the Mesos agent machines, the tasks scheduled on them needs to be drained. +By specifying the SLA requirements a job can make sure that it has enough instances to +continue operating safely without incurring downtime.</p> + +<blockquote> +<p>SLA is defined as minimum number of active tasks required for a job every duration window. +A task is active if it was in <code>RUNNING</code> state during the last duration window.</p> +</blockquote> + +<p>There is a <a href="#default-sla">default</a> SLA guarantee for +<a href="../../features/multitenancy/#configuration-tiers">preferred</a> tier jobs and it is also possible to +specify <a href="#custom-sla">custom</a> SLA requirements.</p> + +<h2 id="default-sla">Default SLA</h2> + +<p>Aurora guarantees a default SLA requirement for tasks in +<a href="../../features/multitenancy/#configuration-tiers">preferred</a> tier.</p> + +<blockquote> +<p>95% of tasks in a job will be <code>active</code> for every 30 mins.</p> +</blockquote> + +<h2 id="custom-sla">Custom SLA</h2> + +<p>For jobs that require different SLA requirements, Aurora allows jobs to specify their own +SLA requirements via the <code>SlaPolicies</code>. There are 3 different ways to express SLA requirements.</p> + +<h3 id="count-based"><a href="../../reference/configuration/#countslapolicy-objects">Count-based</a></h3> + +<p>For jobs that need a minimum <code>number</code> of instances to be running all the time, +<a href="../../reference/configuration/#countslapolicy-objects"><code>CountSlaPolicy</code></a> +provides the ability to express the minimum number of required active instances (i.e. number of +tasks that are <code>RUNNING</code> for at least <code>duration_secs</code>). For instance, if we have a +<code>replicated-service</code> that has 3 instances and needs at least 2 instances every 30 minutes to be +treated healthy, the SLA requirement can be expressed with a +<a href="../../reference/configuration/#countslapolicy-objects"><code>CountSlaPolicy</code></a> like below,</p> +<pre class="highlight python"><code><span style="background-color: #f8f8f8">Job</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">name</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'replicated-service'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">role</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'www-data'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">instances</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">3</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">sla_policy</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">CountSlaPolicy</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">count</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">2</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">duration_secs</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">1800</span> + <span style="background-color: #f8f8f8">)</span> + <span style="color: #000000;font-weight: bold">...</span> +<span style="background-color: #f8f8f8">)</span> +</code></pre> + +<h3 id="percentage-based"><a href="../../reference/configuration/#percentageslapolicy-objects">Percentage-based</a></h3> + +<p>For jobs that need a minimum <code>percentage</code> of instances to be running all the time, +<a href="../../reference/configuration/#percentageslapolicy-objects"><code>PercentageSlaPolicy</code></a> provides the +ability to express the minimum percentage of required active instances (i.e. percentage of tasks +that are <code>RUNNING</code> for at least <code>duration_secs</code>). For instance, if we have a <code>webservice</code> that +has 10000 instances for handling peak load and cannot have more than 0.1% of the instances down +for every 1 hr, the SLA requirement can be expressed with a +<a href="../../reference/configuration/#percentageslapolicy-objects"><code>PercentageSlaPolicy</code></a> like below,</p> +<pre class="highlight python"><code><span style="background-color: #f8f8f8">Job</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">name</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'frontend-service'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">role</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'www-data'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">instances</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">10000</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">sla_policy</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">PercentageSlaPolicy</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">percentage</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">99.9</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">duration_secs</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #009999">3600</span> + <span style="background-color: #f8f8f8">)</span> + <span style="color: #000000;font-weight: bold">...</span> +<span style="background-color: #f8f8f8">)</span> +</code></pre> + +<h3 id="coordinator-based"><a href="../../reference/configuration/#coordinatorslapolicy-objects">Coordinator-based</a></h3> + +<p>When none of the above methods are enough to describe the SLA requirements for a job, then the SLA +calculation can be off-loaded to a custom service called the <code>Coordinator</code>. The <code>Coordinator</code> needs +to expose an endpoint that will be called to check if removal of a task will affect the SLA +requirements for the job. This is useful to control the number of tasks that undergoes maintenance +at a time, without affected the SLA for the application.</p> + +<p>Consider the example, where we have a <code>storage-service</code> stores 2 replicas of an object. Each replica +is distributed across the instances, such that replicas are stored on different hosts. In addition +a consistent-hash is used for distributing the data across the instances.</p> + +<p>When an instance needs to be drained (say for host-maintenance), we have to make sure that at least 1 of +the 2 replicas remains available. In such a case, a <code>Coordinator</code> service can be used to maintain +the SLA guarantees required for the job.</p> + +<p>The job can be configured with a +<a href="../../reference/configuration/#coordinatorslapolicy-objects"><code>CoordinatorSlaPolicy</code></a> to specify the +coordinator endpoint and the field in the response JSON that indicates if the SLA will be affected +or not affected, when the task is removed.</p> +<pre class="highlight python"><code><span style="background-color: #f8f8f8">Job</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">name</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'storage-service'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">role</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'www-data'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">sla_policy</span> <span style="color: #000000;font-weight: bold">=</span> <span style="background-color: #f8f8f8">CoordinatorSlaPolicy</span><span style="background-color: #f8f8f8">(</span> + <span style="background-color: #f8f8f8">coordinator_url</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'http://coordinator.example.com'</span><span style="background-color: #f8f8f8">,</span> + <span style="background-color: #f8f8f8">status_key</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #d14">'drain'</span> + <span style="background-color: #f8f8f8">)</span> + <span style="color: #000000;font-weight: bold">...</span> +<span style="background-color: #f8f8f8">)</span> +</code></pre> + +<h4 id="coordinator-interface-experimental">Coordinator Interface [Experimental]</h4> + +<p>When a <a href="../../reference/configuration/#coordinatorslapolicy-objects"><code>CoordinatorSlaPolicy</code></a> is +specified for a job, any action that requires removing a task +(such as drains) will be required to get approval from the <code>Coordinator</code> before proceeding. The +coordinator service needs to expose a HTTP endpoint, that can take a <code>task-key</code> param +(<code><cluster>/<role>/<env>/<name>/<instance></code>) and a json body describing the task +details, force maintenance countdown (ms) and other params and return a response json that will +contain the boolean status for allowing or disallowing the task’s removal.</p> + +<h5 id="request">Request:</h5> +<pre class="highlight javascript"><code><span style="background-color: #f8f8f8">POST</span> <span style="color: #000000;font-weight: bold">/</span> + <span style="background-color: #f8f8f8">?</span><span style="background-color: #f8f8f8">task</span><span style="color: #000000;font-weight: bold">=<</span><span style="background-color: #f8f8f8">cluster</span><span style="color: #000000;font-weight: bold">></span><span style="color: #009926">/<role>/</span><span style="color: #000000;font-weight: bold"><</span><span style="background-color: #f8f8f8">env</span><span style="color: #000000;font-weight: bold">></span><span style="color: #009926">/<name>/</span><span style="color: #000000;font-weight: bold"><</span><span style="background-color: #f8f8f8">instance</span><span style="color: #000000;font-weight: bold">></span> + +<span style="background-color: #f8f8f8">{</span> + <span style="color: #d14">"forceMaintenanceCountdownMs"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"604755646"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"task"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"cluster/role/devel/job/1"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"taskConfig"</span><span style="background-color: #f8f8f8">:</span> <span style="background-color: #f8f8f8">{</span> + <span style="color: #d14">"assignedTask"</span><span style="background-color: #f8f8f8">:</span> <span style="background-color: #f8f8f8">{</span> + <span style="color: #d14">"taskId"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"taskA"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"slaveHost"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"a"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"task"</span><span style="background-color: #f8f8f8">:</span> <span style="background-color: #f8f8f8">{</span> + <span style="color: #d14">"job"</span><span style="background-color: #f8f8f8">:</span> <span style="background-color: #f8f8f8">{</span> + <span style="color: #d14">"role"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"role"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"environment"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"devel"</span><span style="background-color: #f8f8f8">,</span> + <span style="color: #d14">"name"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #d14">"job"</span> + <span style="background-color: #f8f8f8">},</span> + <span style="background-color: #f8f8f8">...</span> + <span style="background-color: #f8f8f8">},</span> + <span style="color: #d14">"assignedPorts"</span><span style="background-color: #f8f8f8">:</span> <span style="background-color: #f8f8f8">{</span> + <span style="color: #d14">"http"</span><span style="background-color: #f8f8f8">:</span> <span style="color: #009999">1000</span> + <span style="background-color: #f8f8f8">},</span> + <span style="color: #d14">"instanceId"</span><span style="color: #a61717;background-color: #e3d2d2">:</span> <span style="color: #009999">1</span> + <span style="background-color: #f8f8f8">...</span> + <span style="background-color: #f8f8f8">},</span> + <span style="background-color: #f8f8f8">...</span> + <span style="background-color: #f8f8f8">}</span> +<span style="background-color: #f8f8f8">}</span> +</code></pre> + +<h5 id="response">Response:</h5> +<pre class="highlight json"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb"> + </span><span style="color: #000080">"drain"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="color: #bbbbbb"> +</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb"> +</span></code></pre> + +<p>If Coordinator allows removal of the task, then the taskâs +<a href="../../reference/configuration/#httplifecycleconfig-objects">termination lifecycle</a> +is triggered. If Coordinator does not allow removal, then the request will be retried again in the +future.</p> + +<h4 id="coordinator-actions">Coordinator Actions</h4> + +<p>Coordinator endpoint get its own lock and this is used to serializes calls to the Coordinator. +It guarantees that only one concurrent request is sent to a coordinator endpoint. This allows +coordinators to simply look the current state of the tasks to determine its SLA (without having +to worry about in-flight and pending requests). However if there are multiple coordinators, +maintenance can be done in parallel across all the coordinators.</p> + +<p><em>Note: Single concurrent request to a coordinator endpoint does not translate as exactly-once +guarantee. The coordinator must be able to handle duplicate drain +requests for the same task.</em></p> + +</div> + + </div> + </div> + <div class="container-fluid section-footer buffer"> + <div class="container"> + <div class="row"> + <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> + <ul> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Mailing Lists</a></li> + <li><a href="http://issues.apache.org/jira/browse/AURORA";>Issue Tracking</a></li> + <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> + </ul> + </div> + <div class="col-md-2"><h3>The ASF</h3> + <ul> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + <div class="col-md-6"> + <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/";>Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/";>Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/";>Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/";>Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> + </div> + </div> + </div> + + </body> +</html> Added: aurora/site/source/blog/2018-09-10-aurora-0-21-0-released.md URL: http://svn.apache.org/viewvc/aurora/site/source/blog/2018-09-10-aurora-0-21-0-released.md?rev=1840515&view=auto ============================================================================== --- aurora/site/source/blog/2018-09-10-aurora-0-21-0-released.md (added) +++ aurora/site/source/blog/2018-09-10-aurora-0-21-0-released.md Tue Sep 11 05:28:10 2018 @@ -0,0 +1,74 @@ +--- +layout: post +title: 0.21.0 Released +permalink: /blog/aurora-0-21-0-released/ +published: true +post_author: + display_name: Renan DelValle +tags: Release +--- + +The latest Apache Aurora release, 0.21.0, is now available for +[download](http://aurora.apache.org/downloads/). Here are some highlights in this release: + +- Updated to Mesos 1.5.0. +- Introduce ability for tasks to specify custom SLA requirements via the new `SlaPolicy` structs. + There are 3 different SLA Policies that are currently supported - `CountSlaPolicy`, + `PercentageSlaPolicy` and `CoordinatorSlaPolicy`. SLA policies based on count and percentage + express the required number of `RUNNING` instances as either a count or percentage in addition to + allowing the duration-window for which these requirements have to be satisfied. For applications + that need more control over how SLA is determined, a custom SLA calculator can be configured a.k.a + Coordinator. Any action that can affect SLA, will first check with the Coordinator before + proceeding. + + **IMPORTANT: The storage changes required for this feature will make scheduler + snapshot backwards incompatible. Scheduler will be unable to read snapshot if rolled back to + previous version. If rollback is absolutely necessary, perform the following steps:** + 1. Stop all host maintenance requests via `aurora_admin host_activate`. + 2. Ensure a new snapshot is created by running `aurora_admin scheduler_snapshot <cluster>` + 3. Rollback to previous version + + Note: The `Coordinator` interface required for the `CoordinatorSlaPolicy` is experimental at + this juncture and is bound to change in the future. +- Introduced ability for updates to be 'SLA-aware', or only update instances if it is within SLA, + using the new `sla_aware` field in `UpdateConfig`. See the bullet point above for an explanation + of custom SLA requirements. + + **NOTE**: SLA-aware updates will use the desired config's SLA, not the existing config. + + Three additional scheduler options have been added to support this feature: + + 1. `max_update_action_batch_size (default: 300)`: the number of update actions to process in a + batch. + 2. `sla_aware_kill_retry_min_delay (default: 1mins)`: the minimum amount of time to wait before + retrying an SLA-aware kill (using a truncated binary backoff). + 3. `sla_aware_kill_retry_max_delay (default: 5mins)`: the maximum amount of time to wait before + retrying an SLA-aware kill (using a truncated binary backoff). + +Deprecations and removals: + +- Deprecated the `aurora_admin host_drain` command used for maintenance. With this release the SLA + computations are moved to the scheduler and it is no longer required for the client to compute + SLAs and watch the drains. The scheduler persists any host maintenance request and performs + SLA-aware drain of the tasks, before marking the host as `DRAINED`. So maintenance requests + survive across scheduler fail-overs. Use the newly introduced `aurora_admin sla_host_drain` + to skip the SLA computations on the admin client. +- Removed resource fields (`numCpus`, `ramMb`, `diskMb`) from ResourceAggregate. + +Full release notes are available in the release +[CHANGELOG](https://gitbox.apache.org/repos/asf?p=aurora.git&f=CHANGELOG&hb=rel/0.21.0). + +## Thanks + +Thanks to the 9 contributors who made Apache Aurora 0.21.0 possible: + +* Jing Chen +* Renan DelValle +* Stephan Erb +* Jordan Ly +* David McLaughlin +* John Sirois +* Sameer Parekh +* Reza Motamedi +* Ezequiel Torres Feyuk + Added: aurora/site/source/documentation/0.21.0/additional-resources/presentations.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.21.0/additional-resources/presentations.md?rev=1840515&view=auto ============================================================================== --- aurora/site/source/documentation/0.21.0/additional-resources/presentations.md (added) +++ aurora/site/source/documentation/0.21.0/additional-resources/presentations.md Tue Sep 11 05:28:10 2018 @@ -0,0 +1,80 @@ +# Apache Aurora Presentations +Video and slides from presentations and panel discussions about Apache Aurora. + +_(Listed in date descending order)_ + +<table> + + <tr> + <td><img src="/documentation/0.21.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png" alt="Mesos and Aurora on a Small Scale Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=q5iIqhaCJ_o";>Mesos & Aurora on a Small Scale (Video)</a></strong> + <p>Presented by Florian Pfeiffer</p> + <p>October 8, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon-europe";>#MesosCon Europe 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png" alt="SLA Aware Maintenance for Operators Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=tZ0-SISvCis";>SLA Aware Maintenance for Operators (Video)</a></strong> + <p>Presented by Joe Smith</p> + <p>October 8, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon-europe";>#MesosCon Europe 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png" alt="Shipping Code with Aurora Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=y1hi7K1lPkk";>Shipping Code with Aurora (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>August 20, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon";>#MesosCon 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png" alt="Twitter Production Scale Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=nNrh-gdu9m4";>Twitterâs Production Scale: Mesos and Aurora Operations (Video)</a></strong> + <p>Presented by Joe Smith</p> + <p>August 20, 2015 at <a href="http://events.linuxfoundation.org/events/archive/2015/mesoscon";>#MesosCon 2015</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png" alt="From Monolith to Microservices with Aurora Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=yXkOgnyK4Hw";>From Monolith to Microservices w/ Aurora (Video)</a></strong> + <p>Presented by Thanos Baskous, Tony Dong, Dobromir Montauk</p> + <p>April 30, 2015 at <a href="http://www.meetup.com/Bay-Area-Apache-Aurora-Users-Group/events/221219480/";>Bay Area Apache Aurora Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png" alt="Aurora + Mesos in Practice at Twitter Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=1XYJGX_qZVU";>Aurora + Mesos in Practice at Twitter (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>March 07, 2015 at <a href="http://www.bigeng.io/aurora-mesos-in-practice-at-twitter";>Bigcommerce TechTalk</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/02_28_2015_apache_aurora_thumb.png" alt="Apache Auroraã®å§ããã Slideshow Thumbnail" /></td> + <td><strong><a href="http://www.slideshare.net/zembutsu/apache-aurora-introduction-and-tutorial-osc15tk";>Apache Auroraã®å§ããã (Slides)</a></strong> + <p>Presented by Masahito Zembutsu</p> + <p>February 28, 2015 at <a href="http://www.ospn.jp/osc2015-spring/";>Open Source Conference 2015 Tokyo Spring</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png" alt="Apache Aurora Adopters Panel Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=2Jsj0zFdRlg";>Apache Aurora Adopters Panel (Video)</a></strong> + <p>Panelists Ben Staffin, Josh Adams, Bill Farner, Berk Demir</p> + <p>February 19, 2015 at <a href="http://www.meetup.com/Bay-Area-Mesos-User-Group/events/220279080/";>Bay Area Mesos Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png" alt="Operating Apache Aurora and Mesos at Twitter Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=E4lxX6epM_U";>Operating Apache Aurora and Mesos at Twitter (Video)</a></strong> + <p>Presented by Joe Smith</p> + <p>February 19, 2015 at <a href="http://www.meetup.com/Bay-Area-Mesos-User-Group/events/220279080/";>Bay Area Mesos Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png" alt="Apache Aurora and Mesos at TellApart" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=ZZXtXLvTXAE";>Apache Aurora and Mesos at TellApart (Video)</a></strong> + <p>Presented by Steve Niemitz</p> + <p>February 19, 2015 at <a href="http://www.meetup.com/Bay-Area-Mesos-User-Group/events/220279080/";>Bay Area Mesos Users Group</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/08_21_2014_past_present_future_thumb.png" alt="Past, Present, and Future of the Aurora Scheduler Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=Dsc5CPhKs4o";>Past, Present, and Future of the Aurora Scheduler (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>August 21, 2014 at <a href="http://events.linuxfoundation.org/events/archive/2014/mesoscon";>#MesosCon 2014</a></p></td> + </tr> + <tr> + <td><img src="/documentation/0.21.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png" alt="Introduction to Apache Aurora Video Thumbnail" /></td> + <td><strong><a href="https://www.youtube.com/watch?v=asd_h6VzaJc";>Introduction to Apache Aurora (Video)</a></strong> + <p>Presented by Bill Farner</p> + <p>March 25, 2014 at <a href="https://www.eventbrite.com/e/aurora-and-mesosframeworksmeetup-tickets-10850994617";>Aurora and Mesos Frameworks Meetup</a></p></td> + </tr> +</table> Added: aurora/site/source/documentation/0.21.0/additional-resources/tools.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.21.0/additional-resources/tools.md?rev=1840515&view=auto ============================================================================== --- aurora/site/source/documentation/0.21.0/additional-resources/tools.md (added) +++ aurora/site/source/documentation/0.21.0/additional-resources/tools.md Tue Sep 11 05:28:10 2018 @@ -0,0 +1,24 @@ +# Tools + +Various tools integrate with Aurora. Is there a tool missing? Let us know, or submit a patch to add it! + +* Load-balancing technology used to direct traffic to services running on Aurora: + - [synapse](https://github.com/airbnb/synapse) based on HAProxy + - [aurproxy](https://github.com/tellapart/aurproxy) based on nginx + - [jobhopper](https://github.com/benley/aurora-jobhopper) performs HTTP redirects for easy developer and administrator access + +* RPC libraries that integrate with the Aurora's [service discovery mechanism](../../features/service-discovery/): + - [linkerd](https://linkerd.io/) RPC proxy + - [finagle](https://twitter.github.io/finagle) (Scala) + - [scales](https://github.com/steveniemitz/scales) (Python) + +* Monitoring: + - [collectd-aurora](https://github.com/zircote/collectd-aurora) for cluster monitoring using collectd + - [Prometheus Aurora exporter](https://github.com/tommyulfsparre/aurora_exporter) for cluster monitoring using Prometheus + - [Prometheus service discovery integration](http://prometheus.io/docs/operating/configuration/#zookeeper-serverset-sd-configurations-serverset_sd_config) for discovering and monitoring services running on Aurora + +* Packaging and deployment: + - [aurora-packaging](https://github.com/apache/aurora-packaging), the source of the official Aurora packages + +* Thrift Clients: + - [gorealis](https://github.com/paypal/gorealis) for communicating with the scheduler using Go Added: aurora/site/source/documentation/0.21.0/contributing.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.21.0/contributing.md?rev=1840515&view=auto ============================================================================== --- aurora/site/source/documentation/0.21.0/contributing.md (added) +++ aurora/site/source/documentation/0.21.0/contributing.md Tue Sep 11 05:28:10 2018 @@ -0,0 +1,93 @@ +## Get the Source Code + +First things first, you'll need the source! The Aurora source is available from Apache git: + + git clone https://gitbox.apache.org/repos/asf/aurora + +Read the Style Guides +--------------------- +Aurora's codebase is primarily Java and Python and conforms to the Twitter Commons styleguides for +both languages. + +- [Java Style Guide](https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/styleguide.md) +- [Python Style Guide](https://github.com/twitter/commons/blob/master/src/python/twitter/common/styleguide.md) + +## Find Something to Do + +There are issues in [Jira](https://issues.apache.org/jira/browse/AURORA) with the +["newbie" label](https://issues.apache.org/jira/issues/?jql=project%20%3D%20AURORA%20AND%20labels%20%3D%20newbie%20and%20resolution%3Dunresolved) +that are good starting places for new Aurora contributors; pick one of these and dive in! To assign +a task to yourself, first ask for your JIRA id to be whitelisted by either asking in IRC/Slack or by +emailing [email protected]. Once your JIRA account has been whitelisted you can assign tickets +to yourself. The next step is to prepare your patch and finally post it for review. + +## Getting your ReviewBoard Account + +Go to https://reviews.apache.org and create an account. + +## Setting up your ReviewBoard Environment + +Run `./rbt status`. The first time this runs it will bootstrap and you will be asked to login. +Subsequent runs will cache your login credentials. + +## Submitting a Patch for Review + +Post a review with `rbt`, fill out the fields in your browser and hit Publish. + + ./rbt post -o + +If you're unsure about who to add as a reviewer, you can default to adding Stephan Erb (StephanErb) and +Renan DelValle (rdelvalle). They will take care of finding an appropriate reviewer for the patch. + +Once you've done this, you probably want to mark the associated Jira issue as Reviewable. + +## Updating an Existing Review + +Incorporate review feedback, make some more commits, update your existing review, fill out the +fields in your browser and hit Publish. + + ./rbt post -o -r <RB_ID> + +## Getting Your Review Merged + +If you're not an Aurora committer, one of the committers will merge your change in as described +below. Generally, the last reviewer to give the review a 'Ship It!' will be responsible. + +### Merging Your Own Review (Committers) + +Once you have shipits from the right committers, merge your changes in a single commit and mark +the review as submitted. The typical workflow is: + + git checkout master + git pull origin master + ./rbt patch -c <RB_ID> # Verify the automatically-generated commit message looks sane, + # editing if necessary. + git show master # Verify everything looks sane + git push origin master + ./rbt close <RB_ID> + +Note that even if you're developing using feature branches you will not use `git merge` - each +commit will be an atomic change accompanied by a ReviewBoard entry. + +### Merging Someone Else's Review + +Sometimes you'll need to merge someone else's RB. The typical workflow for this is + + git checkout master + git pull origin master + ./rbt patch -c <RB_ID> + git show master # Verify everything looks sane, author is correct + git push origin master + +Note for committers: while we generally use the commit message generated by `./rbt patch` some +changes are often required: + +1. Ensure the the commit message does not exceed 100 characters per line. +2. Remove the "Testing Done" section. It's generally redundant (can be seen by checking the linked + review) or entirely irrelevant to the commit itself. + +## Cleaning Up + +Your patch has landed, congratulations! The last thing you'll want to do before moving on to your +next fix is to clean up your Jira and Reviewboard. The former of which should be marked as +"Resolved" while the latter should be marked as "Submitted". Added: aurora/site/source/documentation/0.21.0/development/client.md URL: http://svn.apache.org/viewvc/aurora/site/source/documentation/0.21.0/development/client.md?rev=1840515&view=auto ============================================================================== --- aurora/site/source/documentation/0.21.0/development/client.md (added) +++ aurora/site/source/documentation/0.21.0/development/client.md Tue Sep 11 05:28:10 2018 @@ -0,0 +1,148 @@ +Developing the Aurora Client +============================ + +The client is written in Python, and uses the +[Pants](http://pantsbuild.github.io/python-readme.html) build tool. + + +Building and Testing +-------------------- + +Building and testing the client code are both done using Pants. The relevant targets to know about +are: + + * Build a client executable: `./pants binary src/main/python/apache/aurora/client:aurora` + * Test client code: `./pants test src/test/python/apache/aurora/client/cli:cli` + +If you want to build a source distribution of the client, you need to run `./build-support/release/make-python-sdists`. + + +Creating Custom Builds +---------------------- + +There are situations where you may want to plug in custom logic to the Client that may not be +applicable to the open source codebase. Rather than create a whole CLI from scratch, you can +easily create your own custom, drop-in replacement aurora.pex using the pants build tool. + +First, create an AuroraCommandLine implementation as an entry-point for registering customizations: + + from apache.aurora.client.cli.client import AuroraCommandLine + + class CustomAuroraCommandLine(AuroraCommandLine): + """Custom AuroraCommandLine for your needs""" + + @property + def name(self): + return "your-company-aurora" + + @classmethod + def get_description(cls): + return 'Your Company internal Aurora client command line' + + def __init__(self): + super(CustomAuroraCommandLine, self).__init__() + # Add custom plugins.. + self.register_plugin(YourCustomPlugin()) + + def register_nouns(self): + super(CustomAuroraCommandLine, self).register_nouns() + # You can even add new commands / sub-commands! + self.register_noun(YourStartUpdateProxy()) + self.register_noun(YourDeployWorkflowCommand()) + +Secondly, create a main entry point: + + def proxy_main(): + client = CustomAuroraCommandLine() + if len(sys.argv) == 1: + sys.argv.append("-h") + sys.exit(client.execute(sys.argv[1:])) + +Finally, you can wire everything up with a pants BUILD file in your project directory: + + python_binary( + name='aurora', + entry_point='your_company.aurora.client:proxy_main', + dependencies=[ + ':client_lib' + ] + ) + + python_library( + name='client_lib', + sources = [ + 'client.py', + 'custom_plugin.py', + 'custom_command.py', + ], + dependencies = [ + # The Apache Aurora client + # Any other dependencies for your custom code + ], + ) + +Using the same commands to build the client as above (but obviously pointing to this BUILD file +instead), you will have a drop-in replacement aurora.pex file with your customizations. + +Running/Debugging +------------------ + +For manually testing client changes against a cluster, we use [Vagrant](https://www.vagrantup.com/). +To start a virtual cluster, you need to install Vagrant, and then run `vagrant up` for the root of +the aurora workspace. This will create a vagrant host named "devcluster", with a Mesos master, a set +of Mesos agents, and an Aurora scheduler. + +If you have a change you would like to test in your local cluster, you'll rebuild the client: + + vagrant ssh -c 'aurorabuild client' + +Once this completes, the `aurora` command will reflect your changes. + + +Running/Debugging in PyCharm +----------------------------- + +It's possible to use PyCharm to run and debug both the client and client tests in an IDE. In order +to do this, first run: + + build-support/python/make-pycharm-virtualenv + +This script will configure a virtualenv with all of our Python requirements. Once the script +completes it will emit instructions for configuring PyCharm: + + Your PyCharm environment is now set up. You can open the project root + directory with PyCharm. + + Once the project is loaded: + - open project settings + - click 'Project Interpreter' + - click the cog in the upper-right corner + - click 'Add Local' + - select 'build-support/python/pycharm.venv/bin/python' + - click 'OK' + +### Running/Debugging Tests +After following these instructions, you should now be able to run/debug tests directly from the IDE +by right-clicking on a test (or test class) and choosing to run or debug: + +[](../images/debug-client-test.png) + +If you've set a breakpoint, you can see the run will now stop and let you debug: + +[](../images/debugging-client-test.png) + +### Running/Debugging the Client +Actually running and debugging the client is unfortunately a bit more complex. You'll need to create +a Run configuration: + +* Go to Run â Edit Configurations +* Click the + icon to add a new configuration. +* Choose python and name the configuration 'client'. +* Set the script path to `/your/path/to/aurora/src/main/python/apache/aurora/client/cli/client.py` +* Set the script parameters to the command you want to run (e.g. `job status <job key>`) +* Expand the Environment section and click the ellipsis to add a new environment variable +* Click the + at the bottom to add a new variable named AURORA_CONFIG_ROOT whose value is the + path where the your cluster configuration can be found. For example, to talk to the scheduler + running in the vagrant image, it would be set to `/your/path/to/aurora/examples/vagrant` (this + is the directory where our example clusters.json is found). +* You should now be able to run and debug this configuration!
