Modified: mesos/site/source/documentation/latest.html.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest.html.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest.html.md (original) +++ mesos/site/source/documentation/latest.html.md Thu Jul 30 17:12:19 2015 @@ -24,6 +24,7 @@ layout: documentation * [Network Monitoring and Isolation](/documentation/latest/network-monitoring/) * [Slave Recovery](/documentation/latest/slave-recovery/) for doing seamless upgrades. * [Tools](/documentation/latest/tools/) for setting up and running a Mesos cluster. +* [SSL](/documentation/latest/mesos-ssl/) for enabling and enforcing SSL communication. ## Running Mesos Frameworks
Modified: mesos/site/source/documentation/latest/authorization.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/authorization.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/authorization.md (original) +++ mesos/site/source/documentation/latest/authorization.md Thu Jul 30 17:12:19 2015 @@ -46,7 +46,7 @@ For example, when a framework (re-)regis Similarly, when a framework launches a task(s), "run_tasks" ACLs are checked to see if the framework (`FrameworkInfo.principal`) is authorized to run the task/executor as the given `user`. If not authorized, the launch is rejected and the framework gets a TASK_LOST. -In the same vein, when a user/principal attempts to shutdown a framework through the "/shutdown" HTTP endpoint on the master, "shutdown_frameworks" ACLs are checked to see if the `principal` is authorized to shutdown the given framework. If not authorized, the shutdown is rejected and the user receives an `Unauthorized` HTTP response. +In the same vein, when a user/principal attempts to shutdown a framework through the "/teardown" HTTP endpoint on the master, "shutdown_frameworks" ACLs are checked to see if the `principal` is authorized to shutdown the given framework. If not authorized, the shutdown is rejected and the user receives an `Unauthorized` HTTP response. There are couple of important things to note: @@ -151,7 +151,7 @@ There are couple of important things to } -8. Only `ops` principal can shutdown any frameworks through "/shutdown" HTTP endpoint. +8. Only `ops` principal can shutdown any frameworks through "/teardown" HTTP endpoint. { "permissive" : false, Modified: mesos/site/source/documentation/latest/committers.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/committers.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/committers.md (original) +++ mesos/site/source/documentation/latest/committers.md Thu Jul 30 17:12:19 2015 @@ -4,9 +4,15 @@ layout: documentation # Committers +An Apache Mesos committer is a contributor who has been given write access to the Apache Mesos code repository and related Apache infrastructure. In the Mesos project, each committer is also a voting member of the PMC. + +## Becoming a committer + +Every new committer has to be proposed by a current committer and then voted in by the members of the Mesos PMC. For details about this process and for candidate requirements see the general [Apache guidelines for assessing new candidates for committership](https://community.apache.org/newcommitter.html). Candidates prepare for their nomination as committer by contributing to the Mesos project and its community, by acting according to the [Apache Way](http://theapacheway.com), and by generally following the path [from contributor to committer](https://community.apache.org/contributors/) for Apache projects. Specifically for the Mesos project, you can make use of the [Apache Mesos Committer Candidate Checklist](https://community.apache.org/committer-candidate-checklist/) for suggestions of what kind of contributions and demonstrated behaviors can be instrumental, and to keep track of your progress. + ## Current Committers -We'd like to thank the following committers to the Apache Mesos project who have helped get the project to where it is today. Committers are voted on by members of the Mesos PMC. This list might be stale, the canonical list is located on [Apache's website](http://people.apache.org/committers-by-project.html#mesos). +We'd like to thank the following committers to the Apache Mesos project who have helped get the project to where it is today. This list might be stale, the canonical list is located on [Apache's website](http://people.apache.org/committers-by-project.html#mesos). <table class="table table-hover table-condensed"> <thead> Modified: mesos/site/source/documentation/latest/getting-started.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/getting-started.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/getting-started.md (original) +++ mesos/site/source/documentation/latest/getting-started.md Thu Jul 30 17:12:19 2015 @@ -10,8 +10,8 @@ There are different ways you can get Mes 1. Download the latest stable release from [Apache](http://mesos.apache.org/downloads/) (***Recommended***) - $ wget http://www.apache.org/dist/mesos/0.22.1/mesos-0.22.1.tar.gz - $ tar -zxf mesos-0.22.1.tar.gz + $ wget http://www.apache.org/dist/mesos/0.23.0/mesos-0.23.0.tar.gz + $ tar -zxf mesos-0.23.0.tar.gz 2. Clone the Mesos git [repository](https://git-wip-us.apache.org/repos/asf/mesos.git) (***Advanced Users Only***) Modified: mesos/site/source/documentation/latest/mesos-ssl.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/mesos-ssl.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/mesos-ssl.md (original) +++ mesos/site/source/documentation/latest/mesos-ssl.md Thu Jul 30 17:12:19 2015 @@ -2,58 +2,59 @@ layout: documentation --- -#Configuration +# Configuration There is currently only one implementation of the [libprocess socket interface](https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/socket.hpp) that supports SSL. This implementation uses [libevent](https://github.com/libevent/libevent). Specifically it relies on the `libevent-openssl` library that wraps `openssl`. -After building `Mesos 0.23.0` from source, assuming you have installed the required [Dependencies](#Dependencies), you can modify your configure line to enable SSL as follows: +After building Mesos 0.23.0 from source, assuming you have installed the required [Dependencies](#Dependencies), you can modify your configure line to enable SSL as follows: ~~~ ../configure --enable-libevent --enable-ssl ~~~ -#Running +# Running Once you have successfully built and installed your new binaries, here are the environment variables that are applicable to the `Master`, `Slave`, `Framework Scheduler/Executor`, or any `libprocess process`: -#####SSL_ENABLED=(false|0,true|1) [default=false|0] +#### SSL_ENABLED=(false|0,true|1) [default=false|0] Turn on or off SSL. When it is turned off it is the equivalent of default mesos with libevent as the backing for events. All sockets default to the non-SSL implementation. When it is turned on, the default configuration for sockets is SSL. This means outgoing connections will use SSL, and incoming connections will be expected to speak SSL as well. None of the below flags are relevant if SSL is not enabled. -#####SSL_SUPPORT_DOWNGRADE=(false|0,true|1) [default=false|0] -Control whether or not non-SSL connections can be established. If this is enabled __on the accepting side__, then the accepting side will downgrade to a non-SSL socket if the connecting side is attempting to communicate via non-SSL. (e.g. http). See [Upgrading Your Cluster](#Upgrading) for more details. +#### SSL_SUPPORT_DOWNGRADE=(false|0,true|1) [default=false|0] +Control whether or not non-SSL connections can be established. If this is enabled __on the accepting side__, then the accepting side will downgrade to a non-SSL socket if the connecting side is attempting to communicate via non-SSL. (e.g. HTTP). See [Upgrading Your Cluster](#Upgrading) for more details. -#####SSL_CERT_FILE=(path to certificate) -The location of the certificate this binary will present. +#### SSL_CERT_FILE=(path to certificate) +The location of the certificate that will be presented. -#####SSL_KEY_FILE=(path to key) +#### SSL_KEY_FILE=(path to key) The location of the private key used by OpenSSL. -#####SSL_VERIFY_CERT=(false|0,true|1) [default=false|0] +#### SSL_VERIFY_CERT=(false|0,true|1) [default=false|0] Control whether certificates are verified when presented. If this is false, even when a certificate is presented, it will not be verified. When `SSL_REQUIRE_CERT` is true, `SSL_VERIFY_CERT` is overridden and all certificates will be verified _and_ required. -#####SSL_REQUIRE_CERT=(false|0,true|1) [default=false|0] +#### SSL_REQUIRE_CERT=(false|0,true|1) [default=false|0] Enforce that certificates must be presented by connecting clients. This means all connections (including tools hitting endpoints) must present valid certificates in order to establish a connection. -#####SSL_VERIFY_DEPTH=(4) [default=4] +#### SSL_VERIFY_DEPTH=(N) [default=4] The maximum depth used to verify certificates. The default is 4. See the OpenSSL documentation or contact your system administrator to learn why you may want to change this. -#####SSL_CA_DIR=(path to CA directory) +#### SSL_CA_DIR=(path to CA directory) The directory used to find the certificate authority / authorities. You can specify `SSL_CA_DIR` or `SSL_CA_FILE` depending on how you want to restrict your certificate authorization. -#####SSL_CA_FILE=(path to CA file) +#### SSL_CA_FILE=(path to CA file) The file used to find the certificate authority. You can specify `SSL_CA_DIR` or `SSL_CA_FILE` depending on how you want to restrict your certificate authorization. -#####SSL_CIPHERS=(accepted ciphers separated by ':') [default=AES128-SHA:AES256-SHA:RC4-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA] +#### SSL_CIPHERS=(accepted ciphers separated by ':') [default=AES128-SHA:AES256-SHA:RC4-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA] A list of `:`-separated ciphers. Use these if you want to restrict or open up the accepted ciphers for OpenSSL. Read the OpenSSL documentation or contact your system administrators to see whether you want to override the default values. -#####SSL_ENABLE_SSL_V2=(false|0,true|1) [default=false|0] -#####SSL_ENABLE_SSL_V3=(false|0,true|1) [default=false|0] -#####SSL_ENABLE_TLS_V1_0=(false|0,true|1) [default=false|0] -#####SSL_ENABLE_TLS_V1_1=(false|0,true|1) [default=false|0] -#####SSL_ENABLE_TLS_V1_2=(false|0,true|1) [default=true|1] -The above switches enable / disable the specified protocols. By default only TLS V1.2 is enabled. The mentality here is to restrict security by default, and force users to open it up explicitly. Many older version of the protocols have known vulnerabilities, so only enable these if you understand the risks fully. +#### SSL_ENABLE_SSL_V3=(false|0,true|1) [default=false|0] +#### SSL_ENABLE_TLS_V1_0=(false|0,true|1) [default=false|0] +#### SSL_ENABLE_TLS_V1_1=(false|0,true|1) [default=false|0] +#### SSL_ENABLE_TLS_V1_2=(false|0,true|1) [default=true|1] +The above switches enable / disable the specified protocols. By default only TLS V1.2 is enabled. SSL V2 is always disabled; there is no switch to enable it. The mentality here is to restrict security by default, and force users to open it up explicitly. Many older version of the protocols have known vulnerabilities, so only enable these if you fully understand the risks. +_SSLv2 is disabled completely because modern versions of OpenSSL disable it using multiple compile time configuration options._ #<a name="Dependencies"></a>Dependencies ### libevent We require the OpenSSL support from libevent. The suggested version of libevent is [`2.0.22-stable`](https://github.com/libevent/libevent/releases/tag/release-2.0.22-stable). As new releases come out we will try to maintain compatibility. + ~~~ // For example, on OSX: brew install libevent @@ -62,15 +63,17 @@ brew install libevent ### OpenSSL We require [OpenSSL](https://github.com/openssl/openssl). There are multiple branches of OpenSSL that are being maintained by the community. Since security requires being vigilant, we recommend reading the release notes for the current releases of OpenSSL and deciding on a version within your organization based on your security needs. Mesos is not too deeply dependent on specific OpenSSL versions, so there is room for you to make security decisions as an organization. Please ensure the `event2` and `openssl` headers are available for building mesos. + ~~~ // For example, on OSX: brew install openssl ~~~ -#<a name="Upgrading"></a>Upgrading Your Cluster +# <a name="Upgrading"></a>Upgrading Your Cluster _There is no SSL specific requirement for upgrading different components in a specific order._ -The recommended strategy is to restart all your components to enable SSL with downgrades support enabled. Once all components have SSL enabled, then do a second restart of all your components to disable downgrades. This strategy will allow each component to be restarted independently at your own convenience with no time restrictions. It will also allow you to try SSL in a subset of your cluster. _Please note:_ While different components in your cluster are serving SSL vs non-SSL traffic, any relative links in the WebUI may be broken. Please see the [WebUI](#WebUI) section for details. Here are sample commands for upgrading your cluster: +The recommended strategy is to restart all your components to enable SSL with downgrades support enabled. Once all components have SSL enabled, then do a second restart of all your components to disable downgrades. This strategy will allow each component to be restarted independently at your own convenience with no time restrictions. It will also allow you to try SSL in a subset of your cluster. __NOTE:__ While different components in your cluster are serving SSL vs non-SSL traffic, any relative links in the WebUI may be broken. Please see the [WebUI](#WebUI) section for details. Here are sample commands for upgrading your cluster: + ~~~ // Restart each component with downgrade support (master, slave, framework): SSL_ENABLED=true SSL_SUPPORT_DOWNGRADE=true SSL_KEY_FILE=<path-to-your-private-key> SSL_CERT_FILE=<path-to-your-certificate> <Any other SSL_* environment variables you may choose> <your-component (e.g. bin/master.sh)> <your-flags> @@ -80,10 +83,10 @@ SSL_ENABLED=true SSL_SUPPORT_DOWNGRADE=f ~~~ The end state is a cluster that is only communicating with SSL. -_Please note:_ Any tools you may use that communicate with your components must be able to talk SSL, or they will be denied. You may choose to maintain `SSL_SUPPORT_DOWNGRADE=true` for some time as you upgrade your internal tooling. The advantage of `SSL_SUPPORT_DOWNGRADE=true` is that all components that speak SSL will do so, while other components may still communicate over insecure channels. +__NOTE:__ Any tools you may use that communicate with your components must be able to speak SSL, or they will be denied. You may choose to maintain `SSL_SUPPORT_DOWNGRADE=true` for some time as you upgrade your internal tooling. The advantage of `SSL_SUPPORT_DOWNGRADE=true` is that all components that speak SSL will do so, while other components may still communicate over insecure channels. -#<a name="WebUI"></a>WebUI -The default Mesos WebUI uses relative links. Some of these links transition between endpoints served by the master and slaves. The WebUI currently does not have enough information to change the 'http' vs 'https' links based on whether the target endpoint is currently being served by an SSL-enabled binary. This may cause certain links in the WebUI to be broken when a cluster is in a transition state between SSL and non-SSL. Any tools that hit these endpoints will still be able to access them as long as they hit the endpoint using the right protocol, or the SSL_SUPPORT_DOWNGRADE option is set to true. +# <a name="WebUI"></a>WebUI +The default Mesos WebUI uses relative links. Some of these links transition between endpoints served by the master and slaves. The WebUI currently does not have enough information to change the 'http' vs 'https' links based on whether the target endpoint is currently being served by an SSL-enabled binary. This may cause certain links in the WebUI to be broken when a cluster is in a transition state between SSL and non-SSL. Any tools that hit these endpoints will still be able to access them as long as they hit the endpoint using the right protocol, or the `SSL_SUPPORT_DOWNGRADE` option is set to true. -###Certificates +### Certificates Most browsers have built in protection that guard transitions between pages served using different certificates. For this reason you may choose to serve both the master and slave endpoints using a common certificate that covers multiple hostnames. If you do not do this, certain links, such as those to slave sandboxes, may seem broken as the browser treats the transition between differing certificates transition as unsafe. Modified: mesos/site/source/documentation/latest/operational-guide.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/operational-guide.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/operational-guide.md (original) +++ mesos/site/source/documentation/latest/operational-guide.md Thu Jul 30 17:12:19 2015 @@ -31,6 +31,13 @@ The following steps indicate how to incr To increase the quorum by N, repeat this process to increment the quorum size N times. +NOTE: Currently, moving out of a single master setup requires wiping the replicated log +state and starting fresh. This will wipe all persistent data (e.g. slaves, maintenance +information, quota information, etc). To move from 1 master to 3 masters: + +1. Stop the standalone master. +2. Remove the replicated log data (`replicated_log` under the `--work_dir`). +3. Start the original master and two new masters with `--quorum=2` ### Decreasing the quorum size @@ -42,6 +49,5 @@ The following steps indicate how to decr To decrease the quorum by N, repeat this process to decrement the quorum size N times. - ### Replacing a master Please see the NOTE section above. So long as the failed master is guaranteed to not re-join the ensemble, it is safe to start a new master _with an empty log_ and allow it to catch up. Modified: mesos/site/source/documentation/latest/oversubscription.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/oversubscription.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/oversubscription.md (original) +++ mesos/site/source/documentation/latest/oversubscription.md Thu Jul 30 17:12:19 2015 @@ -1,4 +1,6 @@ ---- layout: documentation --- +--- +layout: documentation +--- # Oversubscription @@ -48,7 +50,7 @@ resources such as cpu shares, bandwidth, the regular launchTasks() API. To safe-guard frameworks that are not designed to deal with preemption, only frameworks registering with the `REVOCABLE_RESOURCES` capability set in its framework info will receive offers -with revocable resources. Further more, recovable resources cannot be +with revocable resources. Further more, revocable resources cannot be dynamically reserved and persistent volumes should not be created on revocable disk resources. @@ -92,8 +94,8 @@ instructions how to configure Mesos for ### Launching tasks using revocable resources -Launching tasks using recovable resources is done through the existing -`launchTasks` API. Revocable resources will have the `recovable` field set. See +Launching tasks using revocable resources is done through the existing +`launchTasks` API. Revocable resources will have the `revocable` field set. See below for an example offer with regular and revocable resources. ~~~{.json} @@ -301,5 +303,5 @@ The `fixed` resource estimator is enable In the example above, a fixed amount of 14 cpus will be offered as revocable resources. -To select custom a resource estimator and QoS controller, please refer to the +To install a custom resource estimator and QoS controller, please refer to the [modules documentation](/documentation/latest/modules/). Modified: mesos/site/source/documentation/latest/reconciliation.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/reconciliation.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/reconciliation.md (original) +++ mesos/site/source/documentation/latest/reconciliation.md Thu Jul 30 17:12:19 2015 @@ -40,6 +40,15 @@ task state reconciliation. ## Task Reconciliation +Mesos provides two forms of reconciliation: + +* "Explicit" reconciliation: the scheduler sends some of its non-terminal +tasks and the master responds with the latest state for each task, if +possible. +* "Implicit" reconciliation: the scheduler sends an empty list of tasks +and the master responds with the latest state for all currently known +non-terminal tasks. + **Tasks must be reconciled explicitly by the framework after a failure.** This is because the scheduler driver does not persist any task information. @@ -50,6 +59,7 @@ framework. So, for now, let's look at how one needs to implement task state reconciliation in a framework scheduler. + ### API Frameworks send a list of `TaskStatus` messages to the master: @@ -73,25 +83,37 @@ slaves that are transitioning between st ### Algorithm -The technique for performing reconciliation should reconcile all non-terminal -tasks, until an update is received for each task, using exponential backoff: +This technique for explicit reconciliation reconciles all non-terminal tasks, +until an update is received for each task, using exponential backoff to retry +tasks that remain unreconciled. Retries are needed because the master temporarily +may not be able to reply for a particular task. For example, during master +failover the master must re-register all of the slaves to rebuild its +set of known tasks (this process can take minutes for large clusters, and +is bounded by the `--slave_reregister_timeout` flag on the master). + +Steps: 1. let `start = now()` 2. let `remaining = { T in tasks | T is non-terminal }` 3. Perform reconciliation: `reconcile(remaining)` 4. Wait for status updates to arrive (use truncated exponential backoff). For each update, note the time of arrival. -5. let `remaining = { T in remaining | T.last_update_arrival() < start }` +5. let `remaining = { T ϵ remaining | T.last_update_arrival() < start }` 6. If `remaining` is non-empty, go to 3. This reconciliation algorithm **must** be run after each (re-)registration. +Implicit reconciliation (passing an empty list) should also be used +periodically, as a defense against data loss in the framework. Unless a +strict registry is in use on the master, its possible for tasks to resurrect +from a LOST state (without a strict registry the master does not enforce +slave removal across failovers). When an unknown task is encountered, the +scheduler should kill or recover the task. + Notes: * When waiting for updates to arrive, **use a truncated exponential backoff**. This will avoid a snowball effect in the case of the driver or master being backed up. -* Implicit reconciliation (passing an empty list) can also be used -periodically, As a defense against data loss in the framework. * It is beneficial to ensure that only 1 reconciliation is in progress at a time, to avoid a snowball effect in the face of many re-registrations. If another reconciliation should be started while one is in-progress, Modified: mesos/site/source/documentation/latest/release-guide.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/release-guide.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/release-guide.md (original) +++ mesos/site/source/documentation/latest/release-guide.md Thu Jul 30 17:12:19 2015 @@ -175,6 +175,8 @@ This guide describes the process of doin 2. Write a blog post announcing the new release and its features and major bug fixes. +3. Update the Getting Started guide to use the latest release link. + ## Remove old releases from svn Per the guidelines [when to archive](http://www.apache.org/dev/release.html#when-to-archive), we should only keep the latest release in each version under development. Modified: mesos/site/source/documentation/latest/upgrades.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/upgrades.md?rev=1693459&r1=1693458&r2=1693459&view=diff ============================================================================== --- mesos/site/source/documentation/latest/upgrades.md (original) +++ mesos/site/source/documentation/latest/upgrades.md Thu Jul 30 17:12:19 2015 @@ -8,6 +8,10 @@ This document serves as a guide for user ## Upgrading from 0.22.x to 0.23.x +**NOTE** The 'stats.json' endpoints for masters and slaves have been removed. Please use the 'metrics/snapshot' endpoints instead. + +**NOTE** The '/master/shutdown' endpoint is deprecated in favor of the new '/master/teardown' endpoint. + **NOTE** In order to enable decorator modules to remove metadata (environment variables or labels), we changed the meaning of the return value for decorator hooks in Mesos 0.23.0. Please refer to the modules documentation for more details. **NOTE** Slave ping timeouts are now configurable on the master via `--slave_ping_timeout` and `--max_slave_ping_timeouts`. Slaves should be upgraded to 0.23.x before changing these flags.
