Added: mesos/site/source/documentation/latest/modules.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/modules.md?rev=1638021&view=auto ============================================================================== --- mesos/site/source/documentation/latest/modules.md (added) +++ mesos/site/source/documentation/latest/modules.md Tue Nov 11 04:11:00 2014 @@ -0,0 +1,360 @@ +--- +layout: documentation +--- + +# Mesos Modules + +Experimental support for Mesos modules was introduced in Mesos 0.21.0. + +### Disclaimer + +- Use and development of Mesos modules is at own risk! Only graced modules +(modules that are part of Mesos distribution) are maintained by the Mesos +project. + +- Please direct all questions to the relevant module writer and/or write to [email protected]. Questions related to modules sent to user and dev list +will be redirected to the modules list. + + +## What are Mesos modules? +Mesos modules provide a way to easily extend inner workings of Mesos by creating +and using shared libraries that are loaded on demand. Modules can be used to +customize Mesos without having to recompiling/relinking for each specific use +case. Modules can isolate external dependencies into separate libraries, thus +resulting into a smaller Mesos core. +Modules also make it easy to experiment with new features. +For example, imagine loadable allocators that contain a VM (Lua, Python, ...) +which makes it possible to try out new allocator algorithms written in +scripting languages without forcing those dependencies into the project. +Finally, modules provide an easy way for third parties to easily extend Mesos +without having to know all the internal details. + + +## Invoking Mesos modules + +The command-line flag `--modules` is available for Mesos master, slave, and +tests to specify a list of modules to be loaded and be available to the internal +subsystems. + +Use `--modules=filepath` to specify the list of modules via a +file containing a JSON formatted string. 'filepath' can be +of the form 'file:///path/to/file' or '/path/to/file'. + +Use `--modules="{...}"` to specify the list of modules inline. + + +### Example JSON strings: + +1. Load a library `libfoo.so` with two modules `org_apache_mesos_bar` and + `org_apache_mesos_baz`. + + ``` + { + "libraries": [ + { + "file": "/path/to/libfoo.so", + "modules": [ + { + "name": "org_apache_mesos_bar", + }, + { + "name": "org_apache_mesos_baz" + } + ] + } + ] + } + ``` + +2. Load the module `org_apache_mesos_bar` from the library `foo` and pass + the command-line argument `X` with value `Y` (module `org_apache_mesos_baz` + is loaded without any command-line parameters): + + ``` + { + "libraries": [ + { + "name": "foo", + "modules": [ + { + "name": "org_apache_mesos_bar" + "parameters": [ + { + "key": "X", + "value": "Y", + } + ] + }, + { + "name": "org_apache_mesos_bar" + } + ] + } + ] + } + ``` + +3. Specifying modules inline: + + ``` + --modules='{"libraries":[{"file":"/path/to/libfoo.so", "modules":[{"name":"org_apache_mesos_bar"}]}]}' + ``` + +### Library names + +For each library, at least one of the "file" or "path" parameter must be +specified. The "file" parameter may refer to a filename (e.g., "libfoo.so"), a +relative path (e.g., "myLibs/libfoo.so"), or an absolute path (e.g., +"/home/mesos/lib/libfoo.so"). The "name" parameter refers to a library name +(e.g., "foo"). If "name" is specified, it is automatically expanded to a proper +library name for the current platform (e.g., "foo" gets expanded to "libfoo.so" +on Linux, and "libfoo.dylib" on OS X). + +If a library path is not specified in the "file" parameter, the library is +searched in the standard library paths or directories pointed to by the +`LD_LIBRARY_PATH` (`DYLD_LIBRARY_PATH` on OS X) environment variable. + +If both "file" and "name" are specified, "name" is ignored. + +## What kinds of modules are supported? + +Mesos currently only provides Isolator and Authentication modules. Additional +graced modules will be added in the near future. + +## Writing Mesos modules + +### A HelloWorld module + +The following snippet describes the implementation of a module named +"org_apache_mesos_bar" of "TestModule" kind: + +``` + #include <iostream> + #include "test_module.hpp" + + class TestModuleImpl : public TestModule + { + public: + TestModuleImpl() + { + std::cout << "HelloWorld!" << std::endl; + } + + virtual int foo(char a, long b) + { + return a + b; + } + + virtual int bar(float a, double b) + { + return a * b; + } + }; + + static TestModule* create() + { + return new TestModule(); + } + + static bool compatible() + { + return true; + } + + // Declares a module named 'example' of 'TestModule' kind. + // Mesos core binds the module instance pointer as needed. + // The compatible() hook is provided by the module for compatibility checks. + // The create() hook returns an object of type 'TestModule'. + mesos::modules::Module<TestModule> org_apache_mesos_TestModule( + MESOS_MODULE_API_VERSION, + MESOS_VERSION, + "Apache Mesos", + "[email protected]", + "This is a test module.", + compatible, + create); +``` + +### Building a module + + The following assumes that Mesos is installed in the standard location, i.e. + the Mesos dynamic library and header files are available. +``` + g++ -lmesos -fpic -o test_module.o test_module.cpp + $ gcc -shared -o libtest_module.so test_module.o +``` + +### Testing a modules + +Apart from testing the module by hand with explicit use of --modules flag, one +can run the entire mesos test suite with the given module. For example, the +following command will run the mesos test suite with the +`org_apache_mesos_TestCpuIsolator` module selected for isolation: +``` +./bin/mesos-tests.sh --modules="/home/kapil/mesos/isolator-module/modules.json" --isolation="org_apache_mesos_TestCpuIsolator" +``` + +### Module naming convention +Each module name should be unique. Having duplicate module names in the Json +string will cause the process to abort. + +Therefore, we encourage module writers to name their modules according to Java +package naming scheme +(http://docs.oracle.com/javase/tutorial/java/package/namingpkgs.html). + + +For example: + +<table> +<tr> +<th> Module Name </th> <th> Module Domain name </th> <th> Module Symbol Name </th> +</tr> + +<tr> +<td> foobar </td> <td> mesos.apache.org </td> <td> org_apache_mesos_foobar </td> +</tr> + +<tr> +<td> barBaz </td> <td> example.com </td> <td> com_example_barBaz </td> +</table> + +In short: +- Keep case of module name. +- Lower case and reverse domain. +- Separate with underscore. +- Do not simply use the kind name as module name. +- Different modules from the same organization still need different names. + + +## Module Versioning and backwards compatibility + +Before loading the above module, a dynamic library that contains the module +needs to be loaded into Mesos. This happens early in Mesos startup code. The +Mesos developer does not need to touch that code when introducing new module +kinds. However, the developer is responsible for registering what versions of +any given module are expected to remain compatible with Mesos as it progresses. +This information is maintained in a table in `src/module/manager.cpp`. It +contains an entry for every possible module kind that Mesos recognizes, each +with a corresponding Mesos release version number. This number needs to be +adjusted by the Mesos developer to reflect the current Mesos version whenever +compatibility between Mesos and modules that get compiled against it gets +broken. Given that module implementation for older Mesos versions can still be +written in the future, this may be impossible to tell and so in doubt it is best +to just bump the required module version to the current Mesos version. But if +one can be reasonably sure, assuming cooperative module developers, that a +certain kind of module will continue to function across several Mesos versions, +the table provides an easy way to specify this. + +For successfully loading the module, the following relationship +must exist between the various versions: + +` kind version <= Library version <= Mesos version` + +<table> +<tr> +<td>Mesos </td> <td>kind version </td> <td> Library </td> <td>Is module loadable </td> <td>Reason </td> +</tr> + +<tr> +<td>0.18.0 </td> <td> 0.18.0 </td> <td> 0.18.0 </td> <td> yes </td> <td> </td> +</tr> + +<tr> +<td>0.29.0 </td> <td> 0.18.0 </td> <td> 0.18.0 </td> <td> yes </td> <td> </td> +</tr> + +<tr> +<td>0.29.0 </td> <td> 0.18.0 </td> <td> 0.21.0 </td> <td> yes </td> <td> </td> +</tr> + +<tr> +<td>0.18.0 </td> <td> 0.18.0 </td> <td> 0.29.0 </td> <td> NO </td> <td> Library compiled against a newer Mesos release. </td> +</tr> + +<tr> +<td>0.29.0 </td> <td> 0.21.0 </td> <td> 0.18.0 </td> <td> NO </td> <td> Module/Library older than the kind version supported by Mesos. </td> +<tr> +</tr> + +<tr> +<td>0.29.0 </td> <td> 0.29.0 </td> <td> 0.18.0 </td> <td> NO </td> <td> Module/Library older than the kind version supported by Mesos. </td> +</tr> +</table> + +## Mesos module API changes + +Record of incompatible changes to the modules API. + +### Version 2 +- Added support for module-specific command-line parameters. +- Changed function signature for create(). + +### Version 1 +Initial version of the modules API. + + +## Appendix: +### JSON Schema: + +``` + { + "type":"object", + "required":false, + "properties":{ + "libraries": { + "type":"array", + "required":false, + "items": + { + "type":"object", + "required":false, + "properties":{ + "file": { + "type":"string", + "required":false + }, + "name": { + "type":"string", + "required":false + }, + "modules": { + "type":"array", + "required":false, + "items": + { + "type":"object", + "required":false, + "properties":{ + "name": { + "type":"string", + "required":true + }, + "parameters": { + "type":"array", + "required":false, + "items": + { + "type":"object", + "required":false, + "properties":{ + "key": { + "type":"string", + "required":true + }, + "value": { + "type":"string", + "required":true + } + } + } + } + } + } + } + } + } + } + } + } +```
Modified: mesos/site/source/documentation/latest/powered-by-mesos.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/powered-by-mesos.md?rev=1638021&r1=1638020&r2=1638021&view=diff ============================================================================== --- mesos/site/source/documentation/latest/powered-by-mesos.md (original) +++ mesos/site/source/documentation/latest/powered-by-mesos.md Tue Nov 11 04:11:00 2014 @@ -14,6 +14,8 @@ layout: documentation * [CloudPhysics](http://cloudphysics.com) * [Conviva](http://www.conviva.com) * [CorvisaCloud](http://www.corvisacloud.com/) +* [CRP-Gabriel Lippmann](http://www.crpgl.lu) +* [Daemon](http://www.daemon.com.au) * [Devicescape](http://www.devicescape.com) * [DueDil](http://www.duedil.com) * [eBay](http://www.ebay.com) @@ -38,6 +40,7 @@ layout: documentation * [Pinkbike](http://www.pinkbike.com) * [ProfitStars](http://www.profitstars.com) * [Qubit](http://www.qubitproducts.com) +* [Revisely](http://revise.ly) * [Sailthru](http://www.sailthru.com) * [Sharethrough](http://www.sharethrough.com) * [Sigmoid Analytics](http://www.sigmoidanalytics.com/) Modified: mesos/site/source/documentation/latest/release-guide.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/release-guide.md?rev=1638021&r1=1638020&r2=1638021&view=diff ============================================================================== --- mesos/site/source/documentation/latest/release-guide.md (original) +++ mesos/site/source/documentation/latest/release-guide.md Tue Nov 11 04:11:00 2014 @@ -10,6 +10,42 @@ This guide describes the process of doin ## Prerequisites +1. Ensure that you have a GPG key or generate a new one, e.g., using `gpg --gen-key`. + +2. Add your GPG public key to the Apache Mesos dist repository in the KEYS file. + - Fetch the svn repository `svn co https://dist.apache.org/repos/dist/release/mesos` + - Append your public key using one of methods described in KEYS, + e.g., `(gpg --list-sigs <your name> && gpg --armor --export <your name>) >> KEYS`. + - Push the commit: `svn ci` + +3. Submit your GPG public key to a keyserver, e.g., [MIT PGP Public Key Server](https://pgp.mit.edu). + +4. Add your GPG fingerprint to your [Apache account](https://id.apache.org/). + +5. Create a Maven settings file (`~/.m2/settings.xml`) for the Apache + servers. Encrypt your Apache password using `mvn --encrypt-password`. +``` +<settings> + <servers> + <server> + <id>apache.snapshots.https</id> + <username>APACHE USERNAME</username> + <password>APACHE ENCRYPTED PASSWORD</password> + </server> + <server> + <id>apache.releases.https</id> + <username>APACHE USERNAME</username> + <password>APACHE ENCRYPTED PASSWORD</password> + </server> + </servers> +</settings> +``` + +6. Use `gpg-agent` to avoid typing your passphrase repeatedly. + + +## Preparation + 1. Go to [Apache Jira](https://issues.apache.org/jira/browse/MESOS) and make sure that the CHANGELOG for the release version is up to date. @@ -25,12 +61,15 @@ This guide describes the process of doin version in [JIRA](https://issues.apache.org/jira/browse/MESOS#selectedTab=com.atlassian.jira.plugin.system.project%3Aversions-panel) and click on the `Release Notes`. Make sure to configure the release notes in text format. -3. If not already done, update and commit 'configure.ac' for the release. +3. If not already done, update and commit `configure.ac` for the release. + +4. Update and commit `docs/configuration.md` to reflect the current state of + the master, slave, and configure flags. -4. Update and commit the `docs/upgrades.md` with instructions about how to upgrade +5. Update and commit `docs/upgrades.md` with instructions about how to upgrade a live cluster from the previous release version to this release version. -5. If this is a major release please write and commit documentation for this feature. +6. If this is a major release please write and commit documentation for this feature. ## Tagging the release candidate @@ -57,6 +96,7 @@ This guide describes the process of doin 4. It is not uncommon to release multiple release candidates, with increasing release candidate version, if there are bugs found. +5. Update to the *next* Mesos version in `configure.ac`: change `AC_INIT([mesos], [X.Y.Z]))` and commit. ## Voting the release candidate Modified: mesos/site/source/documentation/latest/upgrades.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/upgrades.md?rev=1638021&r1=1638020&r2=1638021&view=diff ============================================================================== --- mesos/site/source/documentation/latest/upgrades.md (original) +++ mesos/site/source/documentation/latest/upgrades.md Tue Nov 11 04:11:00 2014 @@ -6,6 +6,19 @@ layout: documentation This document serves as a guide for users who wish to upgrade an existing mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes. +## Upgrading from 0.20.x to 0.21.x + +**NOTE** Disabling slave checkpointing has been deprecated; the slave --checkpoint flag has been deprecated and will be removed in a future release. + +In order to upgrade a running cluster: + +* Install the new master binaries and restart the masters. +* Install the new slave binaries and restart the slaves. +* Upgrade the schedulers by linking the latest native library (mesos jar upgrade not necessary). +* Restart the schedulers. +* Upgrade the executors by linking the latest native library and mesos jar (if necessary). + + ## Upgrading from 0.19.x to 0.20.x. **NOTE**: The Mesos API has been changed slightly in this release. The CommandInfo has been changed (see below), which makes launching a command more flexible. The 'value' field has been changed from _required_ to _optional_. However, it will not cause any issue during the upgrade (since the existing schedulers always set this field).
