Linked more doc pages from home.md. Also made various cleanups and fixes throughout the docs.
Review: https://reviews.apache.org/r/38860 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/447293df Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/447293df Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/447293df Branch: refs/heads/master Commit: 447293df8eead5d936f0c9cc6b8b267cd6dbc7ee Parents: 6d4a5ce Author: Neil Conway <[email protected]> Authored: Tue Sep 29 18:38:37 2015 -0700 Committer: Adam B <[email protected]> Committed: Tue Sep 29 18:38:37 2015 -0700 ---------------------------------------------------------------------- docs/allocation-module.md | 8 ++++---- docs/app-framework-development-guide.md | 12 ++++++------ docs/attributes-resources.md | 2 +- docs/deploy-scripts.md | 13 +++++++++---- docs/fetcher-cache-internals.md | 20 +++++++++++--------- docs/fetcher.md | 22 ++++++++++++---------- docs/home.md | 18 +++++++++++++++--- docs/mesos-c++-style-guide.md | 2 +- docs/oversubscription.md | 6 +++--- docs/persistent-volume.md | 2 +- docs/reconciliation.md | 8 ++++---- docs/scheduler_http_api.md | 2 +- docs/tools.md | 1 + 13 files changed, 69 insertions(+), 47 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/allocation-module.md ---------------------------------------------------------------------- diff --git a/docs/allocation-module.md b/docs/allocation-module.md index 4ba5ba8..1d0f4be 100644 --- a/docs/allocation-module.md +++ b/docs/allocation-module.md @@ -2,13 +2,13 @@ layout: documentation --- -# Mesos Allocation Module +# Mesos Allocation Modules -The logic that the Mesos master uses to determine which frameworks to make resource offers to is encapsulated in the Master's _allocator module_. The allocator is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, priority, etc., or tune the default hierarchical Dominant Resource Fairness algorithm (see [the DRF paper](http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf)). +The logic that the Mesos master uses to determine which frameworks to make resource offers to is encapsulated in the master's _allocator module_. The allocator is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, priority, etc., or tune the default hierarchical Dominant Resource Fairness algorithm (see [the DRF paper](https://www.cs.berkeley.edu/~alig/papers/drf.pdf)). To use a custom allocator in Mesos, one must: -- [Implement](#writing-a-custom-allocator) an `Allocator` interface as defined in `mesos/master/allocator.hpp`, +- [Implement](#writing-a-custom-allocator) the `Allocator` interface as defined in `mesos/master/allocator.hpp`, - [Wrap](#wiring-up-a-custom-allocator) the allocator implementation in a module and load it in the Mesos master. @@ -21,7 +21,7 @@ The default allocator is `HierarchicalDRFAllocatorProcess`, which lives in `$MES Additionally, the built-in hierarchical allocator can be extended without the need to reimplement the entirety of the allocation logic. This is possible through the use of the `Sorter` abstraction. Sorters define the order in which hierarchy layers (e.g. roles or frameworks) should be offered resources by taking "client" objects and some information about those clients and returning an ordered list of clients. -Sorters are implemented in C++ and inherit the `Sorter` class defined in `$MESOS_HOME/src/master/allocator/sorter/sorter.hpp`. The default sorter is `DRFSorter`, which implements fair sharing and can be found in `$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp`. This sorter is capable of expressing priorities by specifying weights in `Sorter::add()`. Each client's share is divided by its weight. For example, a role that has a weight of `2` will be offered twice as many resources as a role with weight `1`. +Sorters are implemented in C++ and inherit the `Sorter` class defined in `$MESOS_HOME/src/master/allocator/sorter/sorter.hpp`. The default sorter is `DRFSorter`, which implements fair sharing and can be found in `$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp`. This sorter is capable of expressing priorities by specifying weights in `Sorter::add()`. Each client's share is divided by its weight. For example, a role that has a weight of 2 will be offered twice as many resources as a role with weight 1. ## Wiring up a custom allocator http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/app-framework-development-guide.md ---------------------------------------------------------------------- diff --git a/docs/app-framework-development-guide.md b/docs/app-framework-development-guide.md index bada7fe..c7aa08d 100644 --- a/docs/app-framework-development-guide.md +++ b/docs/app-framework-development-guide.md @@ -206,18 +206,18 @@ virtual void error(ExecutorDriver* driver, const std::string& message) = 0; ## Install your Framework -You need to put your framework somewhere that all slaves on the cluster can get it from. If you are running HDFS, you can put your executor into HDFS. Then, you tell Mesos where it is via the `ExecutorInfo` parameter of `MesosSchedulerDriver`'s constructor (e.g. see src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a a Protocol Buffer Message class (defined in `include/mesos/mesos.proto`), and you set its URI field to something like "HDFS://path/to/executor/". Also, you can pass the `frameworks_home` configuration option (defaults to: `MESOS_HOME/frameworks`) to your `mesos-slave` daemons when you launch them to specify where all of your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then set `ExecutorInfo` to be a relative path, and the slave will prepend the value of frameworks_home to the relative path provided. +You need to put your framework somewhere that all slaves on the cluster can get it from. If you are running HDFS, you can put your executor into HDFS. Then, you tell Mesos where it is via the `ExecutorInfo` parameter of `MesosSchedulerDriver`'s constructor (e.g. see src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a Protocol Buffer Message class (defined in `include/mesos/mesos.proto`), and you set its URI field to something like "HDFS://path/to/executor/". Also, you can pass the `frameworks_home` configuration option (defaults to: `MESOS_HOME/frameworks`) to your `mesos-slave` daemons when you launch them to specify where all of your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then set `ExecutorInfo` to be a relative path, and the slave will prepend the value of frameworks_home to the relative path provided. Once you are sure that your executors are available to the mesos-slaves, you should be able to run your scheduler, which will register with the Mesos master, and start receiving resource offers! ## Labels -`Labels` can be found in the `TaskInfo`, `DiscoveryInfo` and `TaskStatus`s and -let's framework and module writers use Labels to tag and pass unstructured -information around Mesos. Labels are free-form key-value pairs supplied by the -framework scheduler or label decorator hooks. Below is the protobuf definitions -of labels: +`Labels` can be found in the `FrameworkInfo`, `TaskInfo`, `DiscoveryInfo` and +`TaskStatus` messages; framework and module writers can use Labels to tag and +pass unstructured information around Mesos. Labels are free-form key-value pairs +supplied by the framework scheduler or label decorator hooks. Below is the +protobuf definitions of labels: ~~~{.proto} optional Labels labels = 11; http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/attributes-resources.md ---------------------------------------------------------------------- diff --git a/docs/attributes-resources.md b/docs/attributes-resources.md index 1ea5dde..f712d09 100644 --- a/docs/attributes-resources.md +++ b/docs/attributes-resources.md @@ -28,7 +28,7 @@ The following are the definitions of these types: ## Attributes -Attributes are key value pairs (where value is optional) that Mesos passes along when it sends offers to frameworks. An attribute value supports 3 different *types*: scalar, range or text. +Attributes are key-value pairs (where value is optional) that Mesos passes along when it sends offers to frameworks. An attribute value supports 3 different *types*: scalar, range or text. attributes : attribute ( ";" attribute )* http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/deploy-scripts.md ---------------------------------------------------------------------- diff --git a/docs/deploy-scripts.md b/docs/deploy-scripts.md index 58b3a69..0dce7d8 100644 --- a/docs/deploy-scripts.md +++ b/docs/deploy-scripts.md @@ -2,13 +2,18 @@ layout: documentation --- -# Deploy Scripts +# Deployment Scripts -Mesos includes a set of scripts in `[install-prefix]/sbin` that can be used to deploy it on a cluster. To use these scripts, you need to create two configuration files: `[install-prefix]/var/mesos/deploy/masters`, which should list the hostname(s) of the node(s) you want to be your masters (one per line), and `[install-prefix]/var/mesos/deploy/slaves`, which should contain a list of hostnames for your slaves (one per line). You can then start a cluster with `[install-prefix]/sbin/mesos-start-cluster.sh` and stop it with `[install-prefix]/sbin/mesos-stop-cluster.sh`. +Mesos includes a set of scripts in `[install-prefix]/sbin` that can be used to deploy it on a cluster. To use these scripts, you need to create two configuration files: -It is also possible to set environment variables, ulimits, etc that will affect the master and slave. by editing `[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh`. One particularly useful setting is `LIBPROCESS_IP`, which tells the master and slave binaries which IP address to bind to; in some installations, the default interface that the hostname resolves to is not the machine's external IP address, so you can set the right IP through this variable. Besides the common environment variables of master and slave configured in `[install-prefix/var/mesos/deploy/mesos-deploy-env.sh`, it is also possible to set master specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-master-env.sh`, slave specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-slave-env.sh`. +1. `[install-prefix]/var/mesos/deploy/masters`, which should list the hostname(s) of the node(s) you want to be your masters (one per line), and +2. `[install-prefix]/var/mesos/deploy/slaves`, which should contain a list of hostnames for your slaves (one per line). -Finally, the deploy scripts do not use ZooKeeper by default. If you want to use ZooKeeper (for multiple masters). Please see the [High Availability](high-availability.md) documentation for details. +You can then start a cluster with `[install-prefix]/sbin/mesos-start-cluster.sh` and stop it with `[install-prefix]/sbin/mesos-stop-cluster.sh`. + +It is also possible to set environment variables, ulimits, etc that will affect the master and slave by editing `[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh`. One particularly useful setting is `LIBPROCESS_IP`, which tells the master and slave binaries which IP address to bind to; in some installations, the default interface that the hostname resolves to is not the machine's external IP address, so you can set the right IP through this variable. Besides the common environment variables of master and slave configured in `[install-prefix/var/mesos/deploy/mesos-deploy-env.sh`, it is also possible to set master specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-master-env.sh`, slave specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-slave-env.sh`. + +Finally, the deploy scripts do not use ZooKeeper by default. If you want to configure Mesos to use ZooKeeper to coordinate multiple master nodes, please see the [High Availability](/documentation/latest/high-availability/) documentation for details. ## Notes http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/fetcher-cache-internals.md ---------------------------------------------------------------------- diff --git a/docs/fetcher-cache-internals.md b/docs/fetcher-cache-internals.md index d17b41e..e8a68d1 100644 --- a/docs/fetcher-cache-internals.md +++ b/docs/fetcher-cache-internals.md @@ -4,7 +4,7 @@ layout: documentation # Mesos Fetcher Cache Internals -It assumed that readers of this document are well familiar with the contents of the overview and user guide of the Mesos fetcher in "fetcher.md". The present document makes direct references to notions defined in the former. +It assumed that readers of this document are familiar with the contents of the [Mesos fetcher user guide](/documentation/latest/fetcher/). The present document makes direct references to notions defined in the former. ## Design goals for the initial fetcher cache prototype: @@ -16,6 +16,7 @@ It assumed that readers of this document are well familiar with the contents of 5. Slave recovery: Support slave recovery. For future releases, we foresee additional features: + 1. Automatic refreshing of cache content when a URI's content has changed. 2. Prefetching URIs for subsequent tasks. Prefetching can run in parallel with task execution. @@ -51,7 +52,7 @@ Based on this setup, the main program flow in the fetcher process is concerned w The fetcher process uses a private instance of class Cache to represent what URIs are cached, where the respective cache files are, what stage of processing they are in, and so on. -The main data structure to hold all this information is a hashmap from URI/user combinations to Cache::Entry objects, which each contain information about an individual cache file on disk. These objects are referenced by shared_ptr, because they can be addressed by multiple callbacks on behalf of concurrent fetch attempts while also being held in the hashmap. +The main data structure to hold all this information is a hashmap from URI/user combinations to `Cache::Entry` objects, which each contain information about an individual cache file on disk. These objects are referenced by shared_ptr, because they can be addressed by multiple callbacks on behalf of concurrent fetch attempts while also being held in the hashmap. A cache entry corresponds directly to a cache file on disk throughout the entire life time of the latter, including before and after its existence. It holds all pertinent state to inform about the phase and results of fetching the corresponding URI. @@ -64,6 +65,7 @@ While a cache entry is referenced it cannot be evicted by a the current or any o The two blue states are essentially the same: no cache file exists. The two green disk states on the right are also the same. The figure only depicts what happens from the point of view of one isolated fetch run. Any given cache entry can be referenced simultaniously by another concurrent fetch run. It must not be evicted as long as it is referenced by any fetching activity. We implement this by reference counting. Every cache entry has a reference count field that gets incremented at the beginning of its use by a fetch run and decremented at its end. The latter must happen no matter whether the run has been successful or whether there has been an error. Increments happen when: + - A new cache entry is created. It is immediately referenced. - An existing cache entry's file download is going to be waited for. - An existing cache entry has a resident cache file that is going to be retrieved. @@ -78,16 +80,15 @@ As menitoned above, the fetcher process' main control flow concerns sorting out  -After going through this procedure for each URI, the fetcher process assembles the gathered list of per-URI actions into a JSON object (FetcherInfo), which is passed to the mesos-fetcher program in an environment variable. The possible fetch actions for a URI are shown at the bottom of the flow chart. After they are determined, the fetcher process invokes mesos-fetcher. +After going through this procedure for each URI, the fetcher process assembles the gathered list of per-URI actions into a JSON object (`FetcherInfo`), which is passed to the mesos-fetcher program in an environment variable. The possible fetch actions for a URI are shown at the bottom of the flow chart. After they are determined, the fetcher process invokes mesos-fetcher. The implementation is oriented at this control flow but its code structure cannot match it directly, because some of these branches must span multiple libprocess continuations. There are two layers of futures, one for each of these phases. -1. Before making fetcher cache items. -- a) Wait for concurrent downloads for pre-existing cache entries -- b) Wait for size fetching combined and then space reservation for new cache entries. - -2. After making fetcher cache items and running mesos-fetcher. -- Complete new cache items with success/failure, which as an important side-effect informs concurrent fetch runs' futures in phase 1/a. + 1. Before making fetcher cache items, + a. Wait for concurrent downloads for pre-existing cache entries. + b. Wait for size fetching combined and then space reservation for new cache entries. + 2. After making fetcher cache items and running mesos-fetcher, + a. Complete new cache items with success/failure, which as an important side-effect informs concurrent fetch runs' futures in phase 1/a. The futures for phase 1 are not shared outside one fetch run. They exclusively guard asynchronous operations for the same fetch run. Their type parameter does not really matter. But each needs to correspond to one URI and eventual fetch item somehow. Multiple variants have been proposed for this. The complexity remains about the same. @@ -105,6 +106,7 @@ Besides, everything touched in 1/a and 1/b needs to be prevented from being cach The resources named "A" and "B" have been fetched with caching into sandbox 1 and 2 below. In the course of this, two cache entries have been created and two files have been downloaded into the cache and named "1" and "2". (Cache file names have unique names that comprise serial numbers.) The next figure illustrates the state after fetching a different cached URI into sandbox 3, which in this case requires evicting a cache-resident file and its entry. Cache eviction removes cache entries in the order of the least recently used cache entries. Steps if "A" was fetched before "B": + 1. Remove the cache entry for "A" from the fetcher process' cache entry table. Its faded depiction is supposed to indicate this. This immediately makes it appear as if the URI has never been cached, even though the cache file is still around. 2. Proceed with fetching "C". This creates a new cache file, which has a different unique name. (The fetcher process remembers in its cache entry which file name belongs to which URI.) http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/fetcher.md ---------------------------------------------------------------------- diff --git a/docs/fetcher.md b/docs/fetcher.md index 638f530..36351a3 100644 --- a/docs/fetcher.md +++ b/docs/fetcher.md @@ -4,8 +4,7 @@ layout: documentation # Mesos Fetcher -Experimental support for the Mesos fetcher _cache_ is introduced in -Mesos 0.23.0. +Mesos 0.23.0 introduced experimental support for the Mesos _fetcher cache_. In this context we loosely regard the term "downloading" as to include copying from local file systems. @@ -15,7 +14,7 @@ from local file systems. The Mesos fetcher is a mechanism to download resources into the sandbox directory of a task in preparation of running the task. As part of a TaskInfo message, the framework ordering the task's execution provides a list of -CommandInfo::URI protobuf values, which becomes the input to the Mesos fetcher. +`CommandInfo::URI` protobuf values, which becomes the input to the Mesos fetcher. By default, each requested URI is downloaded directly into the sandbox directory and repeated requests for the same URI leads to downloading another copy of the @@ -30,7 +29,7 @@ fetcher instance that is used by every kind of containerizer (except the external containerizer variant, which is responsible for its own approach to fetching). -2. The external program "mesos-fetcher" that is invoked by the former. It +2. The external program `mesos-fetcher` that is invoked by the former. It performs all network and disk operations except file deletions and file size queries for cache-internal bookkeeping. It is run as an external OS process in order to shield the slave process from I/O-related hazards. It takes @@ -39,14 +38,14 @@ detailed fetch action descriptions. ## The fetch procedure -Frameworks launch tasks by calling the scheduler driver method launchTasks(), -passing CommandInfo protobuf structures as arguments. This type of structure +Frameworks launch tasks by calling the scheduler driver method `launchTasks()`, +passing `CommandInfo` protobuf structures as arguments. This type of structure specifies (among other things) a command and a list of URIs that need to be "fetched" into the sandbox directory on the the slave node as a precondition for task execution. Hence, when the slave receives a request go launch a task, it calls upon its fetcher, first, to provision the specified resources into the sandbox directory. If fetching fails, the task is not started and the reported -task status is TASK_FAILED. +task status is `TASK_FAILED`. All URIs requested for a given task are fetched sequentially in a single invocation of mesos-fetcher. Here, avoiding download concurrency reduces the @@ -57,7 +56,7 @@ active concurrently due to multiple task launch requests. Before mesos-fetcher is started, the specific fetch actions to be performed for each URI are determined based on the following protobuf structure. (See -"include/mesos/mesos.proto" for more details.) +`include/mesos/mesos.proto` for more details.) message CommandInfo { message URI { @@ -87,7 +86,7 @@ the affected task. If a user name is specified either way, the fetcher first validates that it is in fact a valid user name on the slave. If it is not, fetching fails right here. Otherwise, the sandbox directory is assigned to the specified user as owner -(using chown) at the end of the fetch procedure, before task execution begins. +(using `chown`) at the end of the fetch procedure, before task execution begins. The user name in play has an important effect on caching. Caching is managed on a per-user base, i.e. the combination of user name and "uri" uniquely @@ -170,7 +169,7 @@ the URI. - Local file sizes are probed with systems calls (that follow symbolic links). - HTTP/HTTPS URIs are queried for the "content-length" field in the header. This - is performed by CURL. The reported asset size must be greater than zero or + is performed by `curl`. The reported asset size must be greater than zero or the URI is deemed invalid. - FTP/FTPS is not supported at the time of writing. - Everything else is queried by the local HDFS client. @@ -253,3 +252,6 @@ The following features would be relatively easy to implement additionally. - Extract content while downloading when bypassing the cache. - Prefetch resources for subsequent tasks. This can happen concurrently with running the present task, right after fetching its own resources. + +## Implementation Details +The [Mesos Fetcher Cache Internals](/documentation/latest/fetcher-cache-internals/) describes how the fetcher cache is implemented. http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/home.md ---------------------------------------------------------------------- diff --git a/docs/home.md b/docs/home.md index f8c2b62..86320f6 100644 --- a/docs/home.md +++ b/docs/home.md @@ -11,10 +11,12 @@ layout: documentation ## Running Mesos +* [Getting Started](/documentation/latest/getting-started/) for basic instructions on compiling and installing Mesos. * [Configuration](/documentation/latest/configuration/) for command-line arguments. * [Mesos Containerizer](/documentation/latest/mesos-containerizer/) default containerizer, supports both Linux and POSIX systems. * [Docker Containerizer](/documentation/latest/docker-containerizer/) for launching a Docker image as a Task, or as an Executor. * [External Containerizer](/documentation/latest/external-containerizer/) for custom containerization implementations. +* [Framework Authentication](/documentation/latest/authentication/) * [Framework Authorization](/documentation/latest/authorization/) * [Framework Rate Limiting](/documentation/latest/framework-rate-limiting/) * [Logging and Debugging](/documentation/latest/logging-and-debugging/) for viewing Mesos and framework logs. @@ -27,26 +29,36 @@ layout: documentation * [Tools](/documentation/latest/tools/) for setting up and running a Mesos cluster. * [SSL](/documentation/latest/mesos-ssl/) for enabling and enforcing SSL communication. +## Advanced Features + +* [Attributes and Resources](/documentation/attributes-resources/) for how to describe the slaves that comprise a cluster. +* [Fetcher Cache](/documentation/latest/fetcher/) for how to configure the Mesos fetcher cache. +* [Oversubscription](/documentation/latest/oversubscription/) for how to configure Mesos to take advantage of unused resources to launch "best-effort" tasks. +* [Persistent Volume](/documentation/latest/persistent-volume/) for how to allow tasks to access persistent storage resources. +* [Reservation](/documentation/latest/reservation/) for how to configure Mesos to allow slaves to reserve resources. + ## Running Mesos Frameworks - * [Mesos frameworks](/documentation/latest/mesos-frameworks/) for a list of apps built on top of Mesos, and instructions on how to run them. + * [Mesos frameworks](/documentation/latest/mesos-frameworks/) for a list of apps built on top of Mesos and instructions on how to run them. ## Developing Mesos Frameworks * [Framework Development Guide](/documentation/latest/app-framework-development-guide/) describes how to build applications on top of Mesos. * [Reconciliation](/documentation/latest/reconciliation/) for ensuring a framework's state remains eventually consistent in the face of failures. +* [Scheduler HTTP API](/documentation/latest/scheduler_http_api/) describes the new HTTP API for communication between schedulers and the Mesos master. * [Javadoc](/api/latest/java/) documents the Mesos Java API. * [Doxygen](/api/latest/c++/namespacemesos.html) documents the Mesos C++ API. * [Developer Tools](/documentation/latest/tools/) for hacking on Mesos or writing frameworks. -## Mesos Modules +## Extending Mesos * [Mesos Modules](/documentation/latest/modules/) for specifying Mesos modules for master, slave and tests. +* [Allocation Modules](/documentation/latest/allocation-module/) for how to write custom resource allocators. ## Contributing to Mesos * [Reporting an Issue, Improvement, or Feature](/documentation/latest/reporting-a-bug/) for getting started with JIRA. -* [Submitting a Patch](/documentation/latest/submitting-a-patch/) for getting started with ReviewBoard, and our tooling around it. +* [Submitting a Patch](/documentation/latest/submitting-a-patch/) for getting started with ReviewBoard and our tooling around it. * [Testing Patterns](/documentation/latest/mesos-testing-patterns/) for tips and tricks used in Mesos tests. * [Effective Code Reviewing](/documentation/latest/effective-code-reviewing/) guidelines, tips, and learnings for how to do effective code reviews. * [Engineering Principles and Practices](/documentation/latest/engineering-principles-and-practices/) to serve as a shared set of project-level values for the community. http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/mesos-c++-style-guide.md ---------------------------------------------------------------------- diff --git a/docs/mesos-c++-style-guide.md b/docs/mesos-c++-style-guide.md index d218aab..0b61891 100644 --- a/docs/mesos-c++-style-guide.md +++ b/docs/mesos-c++-style-guide.md @@ -4,7 +4,7 @@ layout: documentation # Mesos C++ Style Guide -The Mesos codebase follows the [Google C++ Style Guide](http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml) with the following differences: +The Mesos codebase follows the [Google C++ Style Guide](http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml) with some notable differences, as described below. Note that the [clang-format](/documentation/latest/clang-format/) tool can be helpful to ensure that some of the mechanical style rules are obeyed. ## Scoping http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/oversubscription.md ---------------------------------------------------------------------- diff --git a/docs/oversubscription.md b/docs/oversubscription.md index b1a6c99..5a31b1f 100644 --- a/docs/oversubscription.md +++ b/docs/oversubscription.md @@ -47,7 +47,7 @@ resources such as cpu shares, bandwidth, etc. ### Frameworks - (5) Frameworks can choose to launch tasks on revocable resources by using - the regular launchTasks() API. To safe-guard frameworks that are not + the regular `launchTasks()` API. To safe-guard frameworks that are not designed to deal with preemption, only frameworks registering with the `REVOCABLE_RESOURCES` capability set in its framework info will receive offers with revocable resources. Further more, revocable resources cannot be @@ -56,7 +56,7 @@ disk resources. ### Task launch - - The revocable task is launched as usual when the runTask request is received + - The revocable task is launched as usual when the `runTask` request is received on the slave. The resources will still be marked as revocable and isolators can take appropriate actions, if certain resources need to be setup differently for revocable and regular tasks. @@ -213,7 +213,7 @@ message QoSCorrection { } ~~~ -## Configuring Mesos for oversubscription +## Configuring oversubscription Five new flags has been added to the slave: http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/persistent-volume.md ---------------------------------------------------------------------- diff --git a/docs/persistent-volume.md b/docs/persistent-volume.md index ae5b0e5..0951ccb 100644 --- a/docs/persistent-volume.md +++ b/docs/persistent-volume.md @@ -2,7 +2,7 @@ layout: documentation --- -# Persistent Volume +# Persistent Volumes Mesos provides a mechanism to create a persistent volume from disk resources. When launching a task, you can create a volume that exists outside http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/reconciliation.md ---------------------------------------------------------------------- diff --git a/docs/reconciliation.md b/docs/reconciliation.md index b50d692..0adfbf1 100644 --- a/docs/reconciliation.md +++ b/docs/reconciliation.md @@ -2,17 +2,17 @@ There's no getting around it, **frameworks on Mesos are distributed systems**. -**Distributed systems must deal with failures**, and partitions (the two are +**Distributed systems must deal with failures** and partitions (the two are indistinguishable from a system's perspective). Concretely, what does this mean for frameworks? Mesos uses an actor-like -**message passing programming model, in which messages are delivered -at-most-once**. (Exceptions to this include task status updates, most of +**message passing** programming model, in which messages are delivered +**at-most-once**. (Exceptions to this include task status updates, most of which are delivered at-least-once through the use of acknowledgements). **The messages passed between the master and the framework are therefore susceptible to be dropped, in the presence of failures**. -When these non-reliable messages are dropped, inconsistent state can arise +When these unreliable messages are dropped, inconsistent state can arise between the framework and Mesos. As a simple example, consider a launch task request sent by a framework. http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/scheduler_http_api.md ---------------------------------------------------------------------- diff --git a/docs/scheduler_http_api.md b/docs/scheduler_http_api.md index cfac510..de6cfc9 100644 --- a/docs/scheduler_http_api.md +++ b/docs/scheduler_http_api.md @@ -471,7 +471,7 @@ In the case of a network partition, the subscription connection between the sche ## Master detection -Mesos has a high-availability mode that uses multiple Mesos masters; one active master (called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the [documentation](high-availability.md). +Mesos has a high-availability mode that uses multiple Mesos masters; one active master (called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the [documentation](/documentation/latest/high-availability.md). Schedulers are expected to make HTTP requests to the leading master. If requests are made to a non-leading master a âHTTP 307 Temporary Redirectâ will be received with the âLocationâ header pointing to the leading master. http://git-wip-us.apache.org/repos/asf/mesos/blob/447293df/docs/tools.md ---------------------------------------------------------------------- diff --git a/docs/tools.md b/docs/tools.md index 09619f2..414decf 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -20,6 +20,7 @@ These tools make it easy to set up and run a Mesos cluster. If you want to hack on Mesos or write a new framework, these tools will help. +* [clang-format](/documentation/latest/clang-format/) to automatically apply some of the style rules dictated by the [Mesos C++ Style Guide](/documentation/latest/mesos-c++-style-guide/). * [Go Bindings and Examples](https://github.com/mesosphere/mesos-go) Write a Mesos framework in Go! Comes with an example scheduler and executor. * [Mesos Framework giter8 Template](https://github.com/mesosphere/scala-sbt-mesos-framework.g8) This is a giter8 template. The result of applying this template is a bare-bones Apache Mesos framework in Scala using SBT for builds and Vagrant for testing on a singleton cluster. * [Scala Hello World](https://gist.github.com/guenter/7471695) A simple Mesos "Hello World": downloads and starts a web server on every node in the cluster.
