Repository: mesos Updated Branches: refs/heads/master fbf5c7e70 -> 34543c768
Updated the allocator related docs. Review: https://reviews.apache.org/r/34545 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/34543c76 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/34543c76 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/34543c76 Branch: refs/heads/master Commit: 34543c768e1f9f1ea24b68129f69fefd507c0065 Parents: fbf5c7e Author: Alexander Rukletsov <[email protected]> Authored: Fri May 29 09:39:47 2015 -0700 Committer: Niklas Q. Nielsen <[email protected]> Committed: Fri May 29 09:39:47 2015 -0700 ---------------------------------------------------------------------- docs/allocation-module.md | 148 +++++++++++++---------------------------- docs/configuration.md | 11 +++ docs/modules.md | 18 ++++- 3 files changed, 73 insertions(+), 104 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/mesos/blob/34543c76/docs/allocation-module.md ---------------------------------------------------------------------- diff --git a/docs/allocation-module.md b/docs/allocation-module.md index bca54b0..5db4aa2 100644 --- a/docs/allocation-module.md +++ b/docs/allocation-module.md @@ -4,123 +4,65 @@ layout: documentation # Mesos Allocation Module -The logic that the Mesos master uses to determine which frameworks to make offer resource offers to is encapsulated in the Master's _allocation module_. The allocation module is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, Dominant Resource Fairness (see [the DRF paper](http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf)), priority, etc. +The logic that the Mesos master uses to determine which frameworks to make resource offers to is encapsulated in the Master's _allocator module_. The allocator is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, priority, etc., or tune the default hierarchical Dominant Resource Fairness algorithm (see [the DRF paper](http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf)). -## Allocation Module API +To use a custom allocator in Mesos, one must: -Mesos is implemented in C++, so allocation modules are implemented in C++, and inherit the @AllocatorProcess@ class defined in @MESOS_HOME/src/master/allocator.hpp@. As of the time of this writing (5/29/13), the API for allocation modules is as follows: +- [Implement](#writing-a-custom-allocator) an `Allocator` interface as defined in `mesos/master/allocator.hpp`, -``` - virtual ~AllocatorProcess() {} - - virtual void initialize( - const Flags& flags, - const process::PID<Master>& master, - const hashmap<std::string, RoleInfo>& roles) = 0; - - virtual void frameworkAdded( - const FrameworkID& frameworkId, - const FrameworkInfo& frameworkInfo, - const Resources& used) = 0; - - virtual void frameworkRemoved( - const FrameworkID& frameworkId) = 0; - - virtual void frameworkActivated( - const FrameworkID& frameworkId, - const FrameworkInfo& frameworkInfo) = 0; - - virtual void frameworkDeactivated( - const FrameworkID& frameworkId) = 0; - - virtual void slaveAdded( - const SlaveID& slaveId, - const SlaveInfo& slaveInfo, - const hashmap<FrameworkID, Resources>& used) = 0; - - virtual void slaveRemoved( - const SlaveID& slaveId) = 0; - - virtual void updateWhitelist( - const Option<hashset<std::string> >& whitelist) = 0; - - virtual void resourcesRequested( - const FrameworkID& frameworkId, - const std::vector<Request>& requests) = 0; - - // Whenever resources are "recovered" in the cluster (e.g., a task - // finishes, an offer is removed because a framework has failed or - // is failing over), or a framework refuses them, the master - // invokes this callback. - virtual void resourcesRecovered( - const FrameworkID& frameworkId, - const SlaveID& slaveId, - const Resources& resources, - const Option<Filters>& filters) = 0; - - // Whenever a framework that has filtered resources wants to revive - // offers for those resources the master invokes this callback. - virtual void offersRevived( - const FrameworkID& frameworkId) = 0; -``` - -The default allocation module is the HierarchicalAllocatorProcess, which can be found in @MESOS_HOME/src/master/hierarchical_allocator_process.hpp@. You can reference this as a starting place if you choose to write your own allocation module. +- [Wrap](#wiring-up-a-custom-allocator) the allocator implementation in a module and load it in the Mesos master. -## Sorter API +## Writing a custom allocator -Additionally, the hierarchical allocator module can be extended without the need to reimplement the entirety of the allocation logic through the use of the @Sorter@ abstraction. +Allocator modules are implemented in C++, the same language in which Mesos is written. They must subclass the `Allocator` interface defined in `mesos/master/allocator.hpp`. However, your implementation can be a C++ proxy, which delegates calls to an actual allocator written in a language of your choice. -Sorters define the order that roles or frameworks should be offered resources in by taking "client" objects and some information about those clients and returning an ordered list of clients. - -Sorters are implemented in C++ and inherit the @Sorter@ class defined in @MESOS_HOME/src/master/sorter.hpp@. As of the time of this writing, the API for Sorters is as follows: - -``` - virtual ~Sorter() {} +The default allocator is `HierarchicalDRFAllocatorProcess`, which lives in `$MESOS_HOME/src/master/allocator/mesos/hierarchical.hpp`. Like most Mesos components, it is actor-based, which means all interface methods are non-blocking and return immediately after putting the corresponding action into the actor's queue. If you would like to design your custom allocator in a similar manner, subclass `MesosAllocatorProcess` from `$MESOS_HOME/src/master/allocator/mesos/allocator.hpp` and wrap your actor-based allocator in `MesosAllocator`. This dispatches calls to the underlying actor and controls its lifetime. You can refer to `HierarchicalDRFAllocatorProcess` as a starting place if you choose to write your own actor-based allocation module. - // Adds a client to allocate resources to. A client - // may be a user or a framework. - virtual void add(const std::string& client, double weight = 1) = 0; - // Removes a client. - virtual void remove(const std::string& client) = 0; +Additionally, the built-in hierarchical allocator can be extended without the need to reimplement the entirety of the allocation logic. This is possible through the use of the `Sorter` abstraction. Sorters define the order in which hierarchy layers (e.g. roles or frameworks) should be offered resources by taking "client" objects and some information about those clients and returning an ordered list of clients. - // Readds a client to the sort after deactivate. - virtual void activate(const std::string& client) = 0; +Sorters are implemented in C++ and inherit the `Sorter` class defined in `$MESOS_HOME/src/master/allocator/sorter/sorter.hpp`. The default sorter is `DRFSorter`, which implements fair sharing and can be found in `$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp`. This sorter is capable of expressing priorities by specifying weights in `Sorter::add()`. Each client's share is divided by its weight. For example, a role that has a weight of `2` will be offered twice as many resources as a role with weight `1`. - // Removes a client from the sort, so it won't get allocated to. - virtual void deactivate(const std::string& client) = 0; +## Wiring up a custom allocator - // Specify that resources have been allocated to the given client. - virtual void allocated(const std::string& client, - const Resources& resources) = 0; +Once a custom allocator has been written, the next step is to override the built-in implementation with your own. This process consists of several steps: - // Specify that resources have been unallocated from the given client. - virtual void unallocated(const std::string& client, - const Resources& resources) = 0; +- Wrap your allocator in a Mesos allocator module, - // Returns the resources that have been allocated to this client. - virtual Resources allocation(const std::string& client) = 0; +- Load this module in Mesos master. - // Add resources to the total pool of resources this - // Sorter should consider. - virtual void add(const Resources& resources) = 0; +An allocator module is a factory function and a module description, as defined in `mesos/module/allocator.hpp`. Assuming the allocation logic is implemented by the `ExternalAllocator` class declared in `external_allocator.hpp`, the following snippet describes the implementation of an allocator module named `ExternalAllocatorModule`: - // Remove resources from the total pool. - virtual void remove(const Resources& resources) = 0; - - // Returns a list of all clients, in the order that they - // should be allocated to, according to this Sorter's policy. - virtual std::list<std::string> sort() = 0; - - // Returns true if this Sorter contains the specified client, - // either active or deactivated. - virtual bool contains(const std::string& client) = 0; - - // Returns the number of clients this Sorter contains, - // either active or deactivated. - virtual int count() = 0; +``` +#include <mesos/master/allocator.hpp> +#include <mesos/module/allocator.hpp> +#include <stout/try.hpp> + +#include "external_allocator.hpp" + +using namespace mesos; +using mesos::master::allocator::Allocator; +using mesos::internal::master::allocator::HierarchicalDRFAllocator; + +static Allocator* createExternalAllocator(const Parameters& parameters) +{ + Try<Allocator*> allocator = ExternalAllocator::create(); + if (allocator.isError()) { + return NULL; + } + + return allocator.get(); +} + +// Declares an ExternalAllocator module named 'ExternalAllocatorModule'. +mesos::modules::Module<Allocator> ExternalAllocatorModule( + MESOS_MODULE_API_VERSION, + MESOS_VERSION, + "Mesos Contributor", + "[email protected]", + "External Allocator module.", + NULL, + createExternalAllocator); ``` -The default @Sorter@ is the DRFSorter, which implements fair sharing and can be found at @MESOS_HOME/src/master/drf_sorter.hpp@. - -For DRF, if weights are specified in Sorter::add, a client's share will be divided by the weight, creating a form of priority. For example, a role that has a weight of 2 will be offered twice as many resources as a role with weight 1. +Refer to the [Mesos Modules documentation](http://mesos.apache.org/documentation/latest/modules/) for instructions how to compile and load a module in Mesos master. http://git-wip-us.apache.org/repos/asf/mesos/blob/34543c76/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 5a41477..4aeb4ad 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -215,6 +215,17 @@ file:///path/to/file (where file contains one of the above)</code></pre> </tr> <tr> <td> + --allocator=VALUE + </td> + <td> + Allocator to use for resource allocation to frameworks. + Use the default <code>HierarchicalDRF</code> allocator, or load + an alternate allocator module using <code>--modules</code>. + (default: HierarchicalDRF) + </td> + </tr> + <tr> + <td> --[no-]authenticate </td> <td> http://git-wip-us.apache.org/repos/asf/mesos/blob/34543c76/docs/modules.md ---------------------------------------------------------------------- diff --git a/docs/modules.md b/docs/modules.md index 835c195..d323f2a 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -117,7 +117,23 @@ If both "file" and "name" are specified, "name" is ignored. ## What kinds of modules are supported? -Here are the various module kinds currently available: +Here are the various module kinds currently available. + +### Allocator + +The Mesos master's _allocator_ periodically determines which framework(s) should be offered the cluster's available resources. Allocator modules enable experimenting with specialized resource allocation algorithms. An example of these could be an allocator that provides a feature currently not supported by the built-in Hierarchical Dominant Resource Fairness allocator, like oversubscription with preemption. + +To load a custom allocator into Mesos master, you need to: + +- Introduce it to the Mesos master by listing it in the `--modules` configuration, + +- Select it as the allocator via the `--allocator` flag. + +For example, the following command will run the Mesos master with `ExternalAllocatorModule` (see [this section](#Example-JSON-strings) for JSON format): + +``` +./bin/mesos-master.sh --work_dir=m/work --modules="file://<modules-including-allocator>.json" --allocator=ExternalAllocatorModule +``` ### Anonymous
