Modified: mesos/site/publish/documentation/oversubscription/index.html URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/oversubscription/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/publish/documentation/oversubscription/index.html (original) +++ mesos/site/publish/documentation/oversubscription/index.html Wed Sep 30 07:05:01 2015 @@ -131,7 +131,7 @@ resources such as cpu shares, bandwidth, <ul> <li>(5) Frameworks can choose to launch tasks on revocable resources by using -the regular launchTasks() API. To safe-guard frameworks that are not +the regular <code>launchTasks()</code> API. To safe-guard frameworks that are not designed to deal with preemption, only frameworks registering with the <code>REVOCABLE_RESOURCES</code> capability set in its framework info will receive offers with revocable resources. Further more, revocable resources cannot be @@ -143,7 +143,7 @@ disk resources.</li> <h3>Task launch</h3> <ul> -<li>The revocable task is launched as usual when the runTask request is received +<li>The revocable task is launched as usual when the <code>runTask</code> request is received on the slave. The resources will still be marked as revocable and isolators can take appropriate actions, if certain resources need to be setup differently for revocable and regular tasks.</li> @@ -300,7 +300,7 @@ the type of action to perform.</p> } </code></pre> -<h2>Configuring Mesos for oversubscription</h2> +<h2>Configuring oversubscription</h2> <p>Five new flags has been added to the slave:</p>
Modified: mesos/site/publish/documentation/persistent-volume/index.html URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/persistent-volume/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/publish/documentation/persistent-volume/index.html (original) +++ mesos/site/publish/documentation/persistent-volume/index.html Wed Sep 30 07:05:01 2015 @@ -81,7 +81,7 @@ <p>See our <a href="/community/">community</a> page for more details.</p> </div> <div class="col-md-8"> - <h1>Persistent Volume</h1> + <h1>Persistent Volumes</h1> <p>Mesos provides a mechanism to create a persistent volume from disk resources. When launching a task, you can create a volume that exists outside Modified: mesos/site/publish/documentation/reconciliation/index.html URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/reconciliation/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/publish/documentation/reconciliation/index.html (original) +++ mesos/site/publish/documentation/reconciliation/index.html Wed Sep 30 07:05:01 2015 @@ -85,17 +85,17 @@ <p>There’s no getting around it, <strong>frameworks on Mesos are distributed systems</strong>.</p> -<p><strong>Distributed systems must deal with failures</strong>, and partitions (the two are +<p><strong>Distributed systems must deal with failures</strong> and partitions (the two are indistinguishable from a system’s perspective).</p> <p>Concretely, what does this mean for frameworks? Mesos uses an actor-like -<strong>message passing programming model, in which messages are delivered -at-most-once</strong>. (Exceptions to this include task status updates, most of +<strong>message passing</strong> programming model, in which messages are delivered +<strong>at-most-once</strong>. (Exceptions to this include task status updates, most of which are delivered at-least-once through the use of acknowledgements). <strong>The messages passed between the master and the framework are therefore susceptible to be dropped, in the presence of failures</strong>.</p> -<p>When these non-reliable messages are dropped, inconsistent state can arise +<p>When these unreliable messages are dropped, inconsistent state can arise between the framework and Mesos.</p> <p>As a simple example, consider a launch task request sent by a framework. Modified: mesos/site/publish/documentation/reporting-a-bug/index.html URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/reporting-a-bug/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/publish/documentation/reporting-a-bug/index.html (original) +++ mesos/site/publish/documentation/reporting-a-bug/index.html Wed Sep 30 07:05:01 2015 @@ -88,7 +88,7 @@ <ul> <li>We track all issues via Apache’s hosted <a href="https://issues.apache.org/jira/browse/MESOS";>JIRA issue tracker</a>, so if you don’t already have an account you’ll need to register for one. It’s quick and easy.</li> <li>A JIRA should be created for every task, feature, bug-fix, etc. This makes it easy to track progress.</li> -<li>If you are planning to work on the ticket, please assign the JIRA issue to yourself before you start working on it. This helps to avoid duplication of work, and alerts anyone who is following that issue. It is highly encouraged to start a discussion with others in the community about how to solve the issue.</li> +<li>If you are planning to work on the ticket, please assign the JIRA issue to yourself before you start working on it. This helps to avoid duplication of work and alerts anyone who is following that issue. It is highly encouraged to start a discussion with others in the community about how to solve the issue. The <a href="/documentation/latest/submitting-a-patch/">contribution guidelines</a> discuss how to contribute to Mesos development in more detail.</li> </ul> Modified: mesos/site/publish/documentation/scheduler_http_api/index.html URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/scheduler_http_api/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/publish/documentation/scheduler_http_api/index.html (original) +++ mesos/site/publish/documentation/scheduler_http_api/index.html Wed Sep 30 07:05:01 2015 @@ -537,7 +537,7 @@ HTTP/1.1 202 Accepted <h2>Master detection</h2> -<p>Mesos has a high-availability mode that uses multiple Mesos masters; one active master (/documentation/latest/called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the <a href="high-availability/">documentation</a>.</p> +<p>Mesos has a high-availability mode that uses multiple Mesos masters; one active master (/documentation/latest/called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the <a href="/documentation/latest/high-availability/">documentation</a>.</p> <p>Schedulers are expected to make HTTP requests to the leading master. If requests are made to a non-leading master a âHTTP 307 Temporary Redirectâ will be received with the âLocationâ header pointing to the leading master.</p> Modified: mesos/site/publish/documentation/submitting-a-patch/index.html URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/submitting-a-patch/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/publish/documentation/submitting-a-patch/index.html (original) +++ mesos/site/publish/documentation/submitting-a-patch/index.html Wed Sep 30 07:05:01 2015 @@ -90,26 +90,46 @@ </ul> -<h3>Before you submit</h3> +<h3>Before you start writing code</h3> <ol> <li><p>Check out the code from the Apache repository via Git. Instructions are on the <a href="http://mesos.apache.org/gettingstarted/";>Getting Started</a> page.</p></li> -<li><p>Join the dev, issues, reviews and builds mailing lists by sending emails to [email protected], [email protected], [email protected] and [email protected] respectively.</p></li> -<li><p>Find a JIRA that is currently unassigned that you want to work on at <a href="https://issues.apache.org/jira/browse/MESOS";>JIRA issue tracker</a>, or create your own (you’ll need a JIRA account for this, see below)!</p> +<li><p>Join the dev, issues, reviews and builds mailing lists by sending emails to <a href="mailto:[email protected]";>[email protected]</a>, <a href="mailto:[email protected]";>[email protected]</a>, <a href="mailto:[email protected]";>[email protected]</a> and <a href="mailto:[email protected]";>[email protected]</a> respectively.</p></li> +<li><p>Find a JIRA issue that is currently unassigned that you want to work on at <a href="https://issues.apache.org/jira/browse/MESOS";>JIRA issue tracker</a>, or create your own (you’ll need a JIRA account for this, see below)!</p> <ol> <li>This could be a JIRA representing a bug (possibly a bug that you encountered and reported, e.g. when trying to build) or a new feature.</li> +<li>Prefer working on issues marked as “<a href="https://issues.apache.org/jira/browse/MESOS-1?jql=project%20%3D%20MESOS%20AND%20status%20%3D%20Accepted";>Accepted</a>”, rather than merely “Open”. If an issue has been accepted, it means at least one Mesos developer thought that the ideas proposed in the issue are worth pursuing further.</li> +<li>Issues marked with the “<a href="https://issues.apache.org/jira/browse/MESOS-1?jql=project%20%3D%20MESOS%20AND%20status%20%3D%20Accepted%20AND%20labels%20%3D%20newbie";>newbie</a>” label can be good candidates for “starter” projects.</li> </ol> </li> <li><p>Assign the JIRA to yourself. To do this, you will need:</p> <ol> <li>An Apache JIRA user account (sign up for one <a href="https://issues.apache.org/jira/secure/Signup!default.jspa";>here</a>).</li> -<li>You need to be added to the list of Mesos “contributors” by a Mesos committer (send email to [email protected]) in order to be assigned (or to assign yourself) to a JIRA issue.</li> +<li>You need to be added to the list of Mesos “contributors” by a Mesos committer (send email to <a href="mailto:[email protected]";>[email protected]</a>) in order to be assigned (or to assign yourself) to a JIRA issue.</li> +</ol> +</li> +<li><p>Formulate a plan for resolving the issue. Propose your plan via comments in the JIRA.</p></li> +<li><p>Find a <strong>shepherd</strong> to collaborate on your patch. A shepherd is a Mesos committer that will work with you to give you feedback on your proposed design, and to eventually commit your change into the Mesos source tree.</p> + +<ol> +<li>To find a shepherd, email the dev mailing list (include a link to your JIRA issue). You can also try asking by adding a comment to your JIRA issue.</li> +<li>You can also find a shepherd by asking the developers on IRC (/documentation/latest/in the <a href="irc://irc.freenode.net/mesos">mesos channel</a> on <a href="https://freenode.net";>Freenode</a>). You can find the current list of committers <a href="committers/">here</a>: a developer that has previously worked on the component you are modifying might be a good candidate shepherd.</li> +</ol> +</li> +</ol> + + +<h3>Create your patch</h3> + +<ol> +<li><p>Create one or more test cases to exercise the bug or the feature (the Mesos team uses <a href="http://en.wikipedia.org/wiki/Test-driven_development";>test-driven development</a>). Before you start coding, make sure these test cases all fail.</p> + +<ol> +<li>The <a href="/documentation/latest/mesos-testing-patterns/">testing patterns</a> page has some suggestions for writing test cases.</li> </ol> </li> -<li><p>Formulate a plan for resolving the issue, propose your plan via comments in the JIRA.</p></li> -<li><p>Create one or more test cases to exercise the bug or the feature (the Mesos team uses <a href="http://en.wikipedia.org/wiki/Test-driven_development";>test-driven development</a>), before you start coding, make sure these test cases all fail.</p></li> <li><p>Make your changes to the code (using whatever IDE/editor you choose) to actually fix the bug or implement the feature.</p> <ol> @@ -127,10 +147,10 @@ </li> </ol> </li> -<li><p>Make sure all of your test cases now pass.</p> +<li><p>Make sure that all of the unit tests pass, including the new test cases you have added: <code>make check</code>.</p> <ol> -<li><code>make check</code></li> +<li>To execute a single unit test (helpful when trying to debug a test case failure), use something like: <code>make check GTEST_FILTER="HTTPTest.Delete"</code>.</li> </ol> </li> <li><p>Make sure to pull in any changes that have been committed to master branch. Using Git, do this via something like:</p> @@ -151,20 +171,23 @@ <li><p>You’re ready to submit your patch for review!</p> <ol> -<li>Log in or create an account at <a href="http://reviews.apache.org";>Apache Review Board</a></li> +<li>Log in or create an account at <a href="http://reviews.apache.org";>Apache Review Board</a>.</li> <li>The easiest (and recommended) way to submit reviews is through <code>post-reviews.py</code> a wrapper around post-review.</li> -<li>First, install RBTools. <a href="https://www.reviewboard.org/docs/rbtools/dev/";>See Instructions</a></li> -<li>Configure post-review. The easiest method is to symlink to the sample config: ln -s support/reviewboardrc .reviewboardrc</li> -<li>Log into Review Board from the command line: run <code>rbt status</code></li> +<li>First, install RBTools. <a href="https://www.reviewboard.org/docs/rbtools/dev/";>See Instructions</a>.</li> +<li>Configure post-review. The easiest method is to symlink to the sample config: <code>ln -s support/reviewboardrc .reviewboardrc</code>.</li> +<li>Log into Review Board from the command line: run <code>rbt status</code>.</li> <li>From your local branch run <code>support/post-reviews.py</code>.</li> <li>Note that <code>post-reviews.py</code> creates a new review for every commit on your branch that is different from the <code>master</code>.</li> -<li>Be sure to add your JIRA issue id (e.g. MESOS-01) to the field labeled “Bugs” (this will automatically link)</li> -<li>Under “Description” in addition to details about your changes, include a description of any documentation pages that need to be added, or are affected by your changes (e.g. did you change or add any configuration options/flags? Did you add a new binary?)</li> +<li>Be sure to add your JIRA issue id (e.g. MESOS-1) to the field labeled “Bugs” (this will automatically link).</li> +<li>Add your shepherd under the “People” field, in the “Reviewers” section. You should also include other Mesos community members who have contributed to the discussion of your proposed change.</li> +<li> Under “Description” in addition to details about your changes, include a description of any documentation pages that need to be added, or are affected by your changes (e.g. did you change or add any configuration options/flags? Did you add a new binary?)</li> +<li> Under “Testing Done”, explain what new tests you have created, what tests were modified, and what procedures you went through to test your changes.</li> </ol> </li> <li><p>Wait for a code review from another Mesos developer via Review Board, address their feedback and upload updated patches until you receive a “Ship It” from a Mesos committer.</p> <ol> +<li>If you don’t receive any feedback, contact your shepherd to remind them. While the committers try their best to provide prompt feedback on proposed changes, they are busy and sometimes a patch gets overlooked.</li> <li>When addressing feedback, adjust your existing commit(s) instead of creating new commits, otherwise <code>post-reviews.py</code> will create a new review (<code>git rebase -i</code> is your friend).</li> <li>Review Board comments should be used for code-specific discussions, and JIRA comments for bigger-picture design discussions.</li> <li>Always respond to each RB comment that you address directly (i.e. each comment can be responded to directly) with either “Done.” or a comment explaining how you addressed it.</li> @@ -179,7 +202,7 @@ <h2>Style Guides</h2> <ul> -<li>For patches to the core, we ask that you follow the <a href="/documentation/latest/mesos-c++-style-guide/">Mesos C++ Style Guide</a></li> +<li>For patches to the core, we ask that you follow the <a href="/documentation/latest/mesos-c++-style-guide/">Mesos C++ Style Guide</a>.</li> </ul> Modified: mesos/site/publish/documentation/tools/index.html URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/tools/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/publish/documentation/tools/index.html (original) +++ mesos/site/publish/documentation/tools/index.html Wed Sep 30 07:05:01 2015 @@ -103,6 +103,7 @@ <p>If you want to hack on Mesos or write a new framework, these tools will help.</p> <ul> +<li><a href="/documentation/latest/clang-format/">clang-format</a> to automatically apply some of the style rules dictated by the <a href="/documentation/latest/mesos-c++-style-guide/">Mesos C++ Style Guide</a>.</li> <li><a href="https://github.com/mesosphere/mesos-go";>Go Bindings and Examples</a> Write a Mesos framework in Go! Comes with an example scheduler and executor.</li> <li><a href="https://github.com/mesosphere/scala-sbt-mesos-framework.g8";>Mesos Framework giter8 Template</a> This is a giter8 template. The result of applying this template is a bare-bones Apache Mesos framework in Scala using SBT for builds and Vagrant for testing on a singleton cluster.</li> <li><a href="https://gist.github.com/guenter/7471695";>Scala Hello World</a> A simple Mesos “Hello World”: downloads and starts a web server on every node in the cluster.</li> Modified: mesos/site/source/documentation/latest.html.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest.html.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest.html.md (original) +++ mesos/site/source/documentation/latest.html.md Wed Sep 30 07:05:01 2015 @@ -11,10 +11,12 @@ layout: documentation ## Running Mesos +* [Getting Started](/documentation/latest/getting-started/) for basic instructions on compiling and installing Mesos. * [Configuration](/documentation/latest/configuration/) for command-line arguments. * [Mesos Containerizer](/documentation/latest/mesos-containerizer/) default containerizer, supports both Linux and POSIX systems. * [Docker Containerizer](/documentation/latest/docker-containerizer/) for launching a Docker image as a Task, or as an Executor. * [External Containerizer](/documentation/latest/external-containerizer/) for custom containerization implementations. +* [Framework Authentication](/documentation/latest/authentication/) * [Framework Authorization](/documentation/latest/authorization/) * [Framework Rate Limiting](/documentation/latest/framework-rate-limiting/) * [Logging and Debugging](/documentation/latest/logging-and-debugging/) for viewing Mesos and framework logs. @@ -27,26 +29,36 @@ layout: documentation * [Tools](/documentation/latest/tools/) for setting up and running a Mesos cluster. * [SSL](/documentation/latest/mesos-ssl/) for enabling and enforcing SSL communication. +## Advanced Features + +* [Attributes and Resources](/documentation/attributes-resources/) for how to describe the slaves that comprise a cluster. +* [Fetcher Cache](/documentation/latest/fetcher/) for how to configure the Mesos fetcher cache. +* [Oversubscription](/documentation/latest/oversubscription/) for how to configure Mesos to take advantage of unused resources to launch "best-effort" tasks. +* [Persistent Volume](/documentation/latest/persistent-volume/) for how to allow tasks to access persistent storage resources. +* [Reservation](/documentation/latest/reservation/) for how to configure Mesos to allow slaves to reserve resources. + ## Running Mesos Frameworks - * [Mesos frameworks](/documentation/latest/mesos-frameworks/) for a list of apps built on top of Mesos, and instructions on how to run them. + * [Mesos frameworks](/documentation/latest/mesos-frameworks/) for a list of apps built on top of Mesos and instructions on how to run them. ## Developing Mesos Frameworks * [Framework Development Guide](/documentation/latest/app-framework-development-guide/) describes how to build applications on top of Mesos. * [Reconciliation](/documentation/latest/reconciliation/) for ensuring a framework's state remains eventually consistent in the face of failures. +* [Scheduler HTTP API](/documentation/latest/scheduler_http_api/) describes the new HTTP API for communication between schedulers and the Mesos master. * [Javadoc](/api/latest/java/) documents the Mesos Java API. * [Doxygen](/api/latest/c++/namespacemesos.html) documents the Mesos C++ API. * [Developer Tools](/documentation/latest/tools/) for hacking on Mesos or writing frameworks. -## Mesos Modules +## Extending Mesos * [Mesos Modules](/documentation/latest/modules/) for specifying Mesos modules for master, slave and tests. +* [Allocation Modules](/documentation/latest/allocation-module/) for how to write custom resource allocators. ## Contributing to Mesos * [Reporting an Issue, Improvement, or Feature](/documentation/latest/reporting-a-bug/) for getting started with JIRA. -* [Submitting a Patch](/documentation/latest/submitting-a-patch/) for getting started with ReviewBoard, and our tooling around it. +* [Submitting a Patch](/documentation/latest/submitting-a-patch/) for getting started with ReviewBoard and our tooling around it. * [Testing Patterns](/documentation/latest/mesos-testing-patterns/) for tips and tricks used in Mesos tests. * [Effective Code Reviewing](/documentation/latest/effective-code-reviewing/) guidelines, tips, and learnings for how to do effective code reviews. * [Engineering Principles and Practices](/documentation/latest/engineering-principles-and-practices/) to serve as a shared set of project-level values for the community. Modified: mesos/site/source/documentation/latest/allocation-module.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/allocation-module.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/allocation-module.md (original) +++ mesos/site/source/documentation/latest/allocation-module.md Wed Sep 30 07:05:01 2015 @@ -2,13 +2,13 @@ layout: documentation --- -# Mesos Allocation Module +# Mesos Allocation Modules -The logic that the Mesos master uses to determine which frameworks to make resource offers to is encapsulated in the Master's _allocator module_. The allocator is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, priority, etc., or tune the default hierarchical Dominant Resource Fairness algorithm (see [the DRF paper](http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf)). +The logic that the Mesos master uses to determine which frameworks to make resource offers to is encapsulated in the master's _allocator module_. The allocator is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, priority, etc., or tune the default hierarchical Dominant Resource Fairness algorithm (see [the DRF paper](https://www.cs.berkeley.edu/~alig/papers/drf.pdf)). To use a custom allocator in Mesos, one must: -- [Implement](#writing-a-custom-allocator) an `Allocator` interface as defined in `mesos/master/allocator.hpp`, +- [Implement](#writing-a-custom-allocator) the `Allocator` interface as defined in `mesos/master/allocator.hpp`, - [Wrap](#wiring-up-a-custom-allocator) the allocator implementation in a module and load it in the Mesos master. @@ -21,7 +21,7 @@ The default allocator is `HierarchicalDR Additionally, the built-in hierarchical allocator can be extended without the need to reimplement the entirety of the allocation logic. This is possible through the use of the `Sorter` abstraction. Sorters define the order in which hierarchy layers (e.g. roles or frameworks) should be offered resources by taking "client" objects and some information about those clients and returning an ordered list of clients. -Sorters are implemented in C++ and inherit the `Sorter` class defined in `$MESOS_HOME/src/master/allocator/sorter/sorter.hpp`. The default sorter is `DRFSorter`, which implements fair sharing and can be found in `$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp`. This sorter is capable of expressing priorities by specifying weights in `Sorter::add()`. Each client's share is divided by its weight. For example, a role that has a weight of `2` will be offered twice as many resources as a role with weight `1`. +Sorters are implemented in C++ and inherit the `Sorter` class defined in `$MESOS_HOME/src/master/allocator/sorter/sorter.hpp`. The default sorter is `DRFSorter`, which implements fair sharing and can be found in `$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp`. This sorter is capable of expressing priorities by specifying weights in `Sorter::add()`. Each client's share is divided by its weight. For example, a role that has a weight of 2 will be offered twice as many resources as a role with weight 1. ## Wiring up a custom allocator Modified: mesos/site/source/documentation/latest/app-framework-development-guide.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/app-framework-development-guide.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/app-framework-development-guide.md (original) +++ mesos/site/source/documentation/latest/app-framework-development-guide.md Wed Sep 30 07:05:01 2015 @@ -206,18 +206,18 @@ virtual void error(ExecutorDriver* drive ## Install your Framework -You need to put your framework somewhere that all slaves on the cluster can get it from. If you are running HDFS, you can put your executor into HDFS. Then, you tell Mesos where it is via the `ExecutorInfo` parameter of `MesosSchedulerDriver`'s constructor (e.g. see src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a a Protocol Buffer Message class (defined in `include/mesos/mesos.proto`), and you set its URI field to something like "HDFS://path/to/executor/". Also, you can pass the `frameworks_home` configuration option (defaults to: `MESOS_HOME/frameworks`) to your `mesos-slave` daemons when you launch them to specify where all of your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then set `ExecutorInfo` to be a relative path, and the slave will prepend the value of frameworks_home to the relative path provided. +You need to put your framework somewhere that all slaves on the cluster can get it from. If you are running HDFS, you can put your executor into HDFS. Then, you tell Mesos where it is via the `ExecutorInfo` parameter of `MesosSchedulerDriver`'s constructor (e.g. see src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a Protocol Buffer Message class (defined in `include/mesos/mesos.proto`), and you set its URI field to something like "HDFS://path/to/executor/". Also, you can pass the `frameworks_home` configuration option (defaults to: `MESOS_HOME/frameworks`) to your `mesos-slave` daemons when you launch them to specify where all of your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then set `ExecutorInfo` to be a relative path, and the slave will prepend the value of frameworks_home to the relative path provided. Once you are sure that your executors are available to the mesos-slaves, you should be able to run your scheduler, which will register with the Mesos master, and start receiving resource offers! ## Labels -`Labels` can be found in the `TaskInfo`, `DiscoveryInfo` and `TaskStatus`s and -let's framework and module writers use Labels to tag and pass unstructured -information around Mesos. Labels are free-form key-value pairs supplied by the -framework scheduler or label decorator hooks. Below is the protobuf definitions -of labels: +`Labels` can be found in the `FrameworkInfo`, `TaskInfo`, `DiscoveryInfo` and +`TaskStatus` messages; framework and module writers can use Labels to tag and +pass unstructured information around Mesos. Labels are free-form key-value pairs +supplied by the framework scheduler or label decorator hooks. Below is the +protobuf definitions of labels: ~~~{.proto} optional Labels labels = 11; Modified: mesos/site/source/documentation/latest/attributes-resources.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/attributes-resources.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/attributes-resources.md (original) +++ mesos/site/source/documentation/latest/attributes-resources.md Wed Sep 30 07:05:01 2015 @@ -28,7 +28,7 @@ The following are the definitions of the ## Attributes -Attributes are key value pairs (where value is optional) that Mesos passes along when it sends offers to frameworks. An attribute value supports 3 different *types*: scalar, range or text. +Attributes are key-value pairs (where value is optional) that Mesos passes along when it sends offers to frameworks. An attribute value supports 3 different *types*: scalar, range or text. attributes : attribute ( ";" attribute )* Modified: mesos/site/source/documentation/latest/deploy-scripts.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/deploy-scripts.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/deploy-scripts.md (original) +++ mesos/site/source/documentation/latest/deploy-scripts.md Wed Sep 30 07:05:01 2015 @@ -2,13 +2,18 @@ layout: documentation --- -# Deploy Scripts +# Deployment Scripts -Mesos includes a set of scripts in `[install-prefix]/sbin` that can be used to deploy it on a cluster. To use these scripts, you need to create two configuration files: `[install-prefix]/var/mesos/deploy/masters`, which should list the hostname(s) of the node(s) you want to be your masters (one per line), and `[install-prefix]/var/mesos/deploy/slaves`, which should contain a list of hostnames for your slaves (one per line). You can then start a cluster with `[install-prefix]/sbin/mesos-start-cluster.sh` and stop it with `[install-prefix]/sbin/mesos-stop-cluster.sh`. +Mesos includes a set of scripts in `[install-prefix]/sbin` that can be used to deploy it on a cluster. To use these scripts, you need to create two configuration files: -It is also possible to set environment variables, ulimits, etc that will affect the master and slave. by editing `[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh`. One particularly useful setting is `LIBPROCESS_IP`, which tells the master and slave binaries which IP address to bind to; in some installations, the default interface that the hostname resolves to is not the machine's external IP address, so you can set the right IP through this variable. Besides the common environment variables of master and slave configured in `[install-prefix/var/mesos/deploy/mesos-deploy-env.sh`, it is also possible to set master specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-master-env.sh`, slave specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-slave-env.sh`. +1. `[install-prefix]/var/mesos/deploy/masters`, which should list the hostname(s) of the node(s) you want to be your masters (one per line), and +2. `[install-prefix]/var/mesos/deploy/slaves`, which should contain a list of hostnames for your slaves (one per line). -Finally, the deploy scripts do not use ZooKeeper by default. If you want to use ZooKeeper (/documentation/latest/for multiple masters). Please see the [High Availability](high-availability/) documentation for details. +You can then start a cluster with `[install-prefix]/sbin/mesos-start-cluster.sh` and stop it with `[install-prefix]/sbin/mesos-stop-cluster.sh`. + +It is also possible to set environment variables, ulimits, etc that will affect the master and slave by editing `[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh`. One particularly useful setting is `LIBPROCESS_IP`, which tells the master and slave binaries which IP address to bind to; in some installations, the default interface that the hostname resolves to is not the machine's external IP address, so you can set the right IP through this variable. Besides the common environment variables of master and slave configured in `[install-prefix/var/mesos/deploy/mesos-deploy-env.sh`, it is also possible to set master specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-master-env.sh`, slave specific environment variables in `[install-prefix]/var/mesos/deploy/mesos-slave-env.sh`. + +Finally, the deploy scripts do not use ZooKeeper by default. If you want to configure Mesos to use ZooKeeper to coordinate multiple master nodes, please see the [High Availability](/documentation/latest/high-availability/) documentation for details. ## Notes Modified: mesos/site/source/documentation/latest/fetcher-cache-internals.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/fetcher-cache-internals.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/fetcher-cache-internals.md (original) +++ mesos/site/source/documentation/latest/fetcher-cache-internals.md Wed Sep 30 07:05:01 2015 @@ -4,7 +4,7 @@ layout: documentation # Mesos Fetcher Cache Internals -It assumed that readers of this document are well familiar with the contents of the overview and user guide of the Mesos fetcher in "fetcher.md". The present document makes direct references to notions defined in the former. +It assumed that readers of this document are familiar with the contents of the [Mesos fetcher user guide](/documentation/latest/fetcher/). The present document makes direct references to notions defined in the former. ## Design goals for the initial fetcher cache prototype: @@ -16,6 +16,7 @@ It assumed that readers of this document 5. Slave recovery: Support slave recovery. For future releases, we foresee additional features: + 1. Automatic refreshing of cache content when a URI's content has changed. 2. Prefetching URIs for subsequent tasks. Prefetching can run in parallel with task execution. @@ -51,7 +52,7 @@ Based on this setup, the main program fl The fetcher process uses a private instance of class Cache to represent what URIs are cached, where the respective cache files are, what stage of processing they are in, and so on. -The main data structure to hold all this information is a hashmap from URI/user combinations to Cache::Entry objects, which each contain information about an individual cache file on disk. These objects are referenced by shared_ptr, because they can be addressed by multiple callbacks on behalf of concurrent fetch attempts while also being held in the hashmap. +The main data structure to hold all this information is a hashmap from URI/user combinations to `Cache::Entry` objects, which each contain information about an individual cache file on disk. These objects are referenced by shared_ptr, because they can be addressed by multiple callbacks on behalf of concurrent fetch attempts while also being held in the hashmap. A cache entry corresponds directly to a cache file on disk throughout the entire life time of the latter, including before and after its existence. It holds all pertinent state to inform about the phase and results of fetching the corresponding URI. @@ -64,6 +65,7 @@ While a cache entry is referenced it can The two blue states are essentially the same: no cache file exists. The two green disk states on the right are also the same. The figure only depicts what happens from the point of view of one isolated fetch run. Any given cache entry can be referenced simultaniously by another concurrent fetch run. It must not be evicted as long as it is referenced by any fetching activity. We implement this by reference counting. Every cache entry has a reference count field that gets incremented at the beginning of its use by a fetch run and decremented at its end. The latter must happen no matter whether the run has been successful or whether there has been an error. Increments happen when: + - A new cache entry is created. It is immediately referenced. - An existing cache entry's file download is going to be waited for. - An existing cache entry has a resident cache file that is going to be retrieved. @@ -78,16 +80,15 @@ As menitoned above, the fetcher process'  -After going through this procedure for each URI, the fetcher process assembles the gathered list of per-URI actions into a JSON object (FetcherInfo), which is passed to the mesos-fetcher program in an environment variable. The possible fetch actions for a URI are shown at the bottom of the flow chart. After they are determined, the fetcher process invokes mesos-fetcher. +After going through this procedure for each URI, the fetcher process assembles the gathered list of per-URI actions into a JSON object (`FetcherInfo`), which is passed to the mesos-fetcher program in an environment variable. The possible fetch actions for a URI are shown at the bottom of the flow chart. After they are determined, the fetcher process invokes mesos-fetcher. The implementation is oriented at this control flow but its code structure cannot match it directly, because some of these branches must span multiple libprocess continuations. There are two layers of futures, one for each of these phases. -1. Before making fetcher cache items. -- a) Wait for concurrent downloads for pre-existing cache entries -- b) Wait for size fetching combined and then space reservation for new cache entries. - -2. After making fetcher cache items and running mesos-fetcher. -- Complete new cache items with success/failure, which as an important side-effect informs concurrent fetch runs' futures in phase 1/a. + 1. Before making fetcher cache items, + a. Wait for concurrent downloads for pre-existing cache entries. + b. Wait for size fetching combined and then space reservation for new cache entries. + 2. After making fetcher cache items and running mesos-fetcher, + a. Complete new cache items with success/failure, which as an important side-effect informs concurrent fetch runs' futures in phase 1/a. The futures for phase 1 are not shared outside one fetch run. They exclusively guard asynchronous operations for the same fetch run. Their type parameter does not really matter. But each needs to correspond to one URI and eventual fetch item somehow. Multiple variants have been proposed for this. The complexity remains about the same. @@ -105,6 +106,7 @@ Besides, everything touched in 1/a and 1 The resources named "A" and "B" have been fetched with caching into sandbox 1 and 2 below. In the course of this, two cache entries have been created and two files have been downloaded into the cache and named "1" and "2". (Cache file names have unique names that comprise serial numbers.) The next figure illustrates the state after fetching a different cached URI into sandbox 3, which in this case requires evicting a cache-resident file and its entry. Cache eviction removes cache entries in the order of the least recently used cache entries. Steps if "A" was fetched before "B": + 1. Remove the cache entry for "A" from the fetcher process' cache entry table. Its faded depiction is supposed to indicate this. This immediately makes it appear as if the URI has never been cached, even though the cache file is still around. 2. Proceed with fetching "C". This creates a new cache file, which has a different unique name. (The fetcher process remembers in its cache entry which file name belongs to which URI.) Modified: mesos/site/source/documentation/latest/fetcher.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/fetcher.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/fetcher.md (original) +++ mesos/site/source/documentation/latest/fetcher.md Wed Sep 30 07:05:01 2015 @@ -4,8 +4,7 @@ layout: documentation # Mesos Fetcher -Experimental support for the Mesos fetcher _cache_ is introduced in -Mesos 0.23.0. +Mesos 0.23.0 introduced experimental support for the Mesos _fetcher cache_. In this context we loosely regard the term "downloading" as to include copying from local file systems. @@ -15,7 +14,7 @@ from local file systems. The Mesos fetcher is a mechanism to download resources into the sandbox directory of a task in preparation of running the task. As part of a TaskInfo message, the framework ordering the task's execution provides a list of -CommandInfo::URI protobuf values, which becomes the input to the Mesos fetcher. +`CommandInfo::URI` protobuf values, which becomes the input to the Mesos fetcher. By default, each requested URI is downloaded directly into the sandbox directory and repeated requests for the same URI leads to downloading another copy of the @@ -30,7 +29,7 @@ fetcher instance that is used by every k external containerizer variant, which is responsible for its own approach to fetching). -2. The external program "mesos-fetcher" that is invoked by the former. It +2. The external program `mesos-fetcher` that is invoked by the former. It performs all network and disk operations except file deletions and file size queries for cache-internal bookkeeping. It is run as an external OS process in order to shield the slave process from I/O-related hazards. It takes @@ -39,14 +38,14 @@ detailed fetch action descriptions. ## The fetch procedure -Frameworks launch tasks by calling the scheduler driver method launchTasks(), -passing CommandInfo protobuf structures as arguments. This type of structure +Frameworks launch tasks by calling the scheduler driver method `launchTasks()`, +passing `CommandInfo` protobuf structures as arguments. This type of structure specifies (among other things) a command and a list of URIs that need to be "fetched" into the sandbox directory on the the slave node as a precondition for task execution. Hence, when the slave receives a request go launch a task, it calls upon its fetcher, first, to provision the specified resources into the sandbox directory. If fetching fails, the task is not started and the reported -task status is TASK_FAILED. +task status is `TASK_FAILED`. All URIs requested for a given task are fetched sequentially in a single invocation of mesos-fetcher. Here, avoiding download concurrency reduces the @@ -57,7 +56,7 @@ active concurrently due to multiple task Before mesos-fetcher is started, the specific fetch actions to be performed for each URI are determined based on the following protobuf structure. (See -"include/mesos/mesos.proto" for more details.) +`include/mesos/mesos.proto` for more details.) message CommandInfo { message URI { @@ -87,7 +86,7 @@ the affected task. If a user name is specified either way, the fetcher first validates that it is in fact a valid user name on the slave. If it is not, fetching fails right here. Otherwise, the sandbox directory is assigned to the specified user as owner -(using chown) at the end of the fetch procedure, before task execution begins. +(using `chown`) at the end of the fetch procedure, before task execution begins. The user name in play has an important effect on caching. Caching is managed on a per-user base, i.e. the combination of user name and "uri" uniquely @@ -170,7 +169,7 @@ the URI. - Local file sizes are probed with systems calls (that follow symbolic links). - HTTP/HTTPS URIs are queried for the "content-length" field in the header. This - is performed by CURL. The reported asset size must be greater than zero or + is performed by `curl`. The reported asset size must be greater than zero or the URI is deemed invalid. - FTP/FTPS is not supported at the time of writing. - Everything else is queried by the local HDFS client. @@ -253,3 +252,6 @@ The following features would be relative - Extract content while downloading when bypassing the cache. - Prefetch resources for subsequent tasks. This can happen concurrently with running the present task, right after fetching its own resources. + +## Implementation Details +The [Mesos Fetcher Cache Internals](/documentation/latest/fetcher-cache-internals/) describes how the fetcher cache is implemented. Modified: mesos/site/source/documentation/latest/mesos-c++-style-guide.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/mesos-c%2B%2B-style-guide.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/mesos-c++-style-guide.md (original) +++ mesos/site/source/documentation/latest/mesos-c++-style-guide.md Wed Sep 30 07:05:01 2015 @@ -4,7 +4,13 @@ layout: documentation # Mesos C++ Style Guide -The Mesos codebase follows the [Google C++ Style Guide](http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml) with the following differences: +The Mesos codebase follows the [Google C++ Style Guide](http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml) with some notable differences, as described below. Note that the [clang-format](/documentation/latest/clang-format/) tool can be helpful to ensure that some of the mechanical style rules are obeyed. + +## Scoping + +### Namespaces +* We avoid `using namespace foo` statements as it is not explicit about which symbols are pulled in, and it can often pull in a lot of symbols, which sometimes lead to conflicts. +* It is OK to use namespace aliases to help pull in sub-namespaces, such as `namespace http = process::http;`. These should only be present at the top of the .cpp file. ## Naming @@ -38,9 +44,6 @@ void Slave::statusUpdate(StatusUpdate up ### Function Names * We use [lowerCamelCase](http://en.wikipedia.org/wiki/CamelCase#Variations_and_synonyms) for function names (Google uses mixed case for regular functions; and their accessors and mutators match the name of the variable). -### Namespace Names -* We do not use namespace aliases. - ## Strings * Strings used in log and error messages should end without a period. Modified: mesos/site/source/documentation/latest/oversubscription.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/oversubscription.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/oversubscription.md (original) +++ mesos/site/source/documentation/latest/oversubscription.md Wed Sep 30 07:05:01 2015 @@ -47,7 +47,7 @@ resources such as cpu shares, bandwidth, ### Frameworks - (5) Frameworks can choose to launch tasks on revocable resources by using - the regular launchTasks() API. To safe-guard frameworks that are not + the regular `launchTasks()` API. To safe-guard frameworks that are not designed to deal with preemption, only frameworks registering with the `REVOCABLE_RESOURCES` capability set in its framework info will receive offers with revocable resources. Further more, revocable resources cannot be @@ -56,7 +56,7 @@ disk resources. ### Task launch - - The revocable task is launched as usual when the runTask request is received + - The revocable task is launched as usual when the `runTask` request is received on the slave. The resources will still be marked as revocable and isolators can take appropriate actions, if certain resources need to be setup differently for revocable and regular tasks. @@ -213,7 +213,7 @@ message QoSCorrection { } ~~~ -## Configuring Mesos for oversubscription +## Configuring oversubscription Five new flags has been added to the slave: Modified: mesos/site/source/documentation/latest/persistent-volume.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/persistent-volume.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/persistent-volume.md (original) +++ mesos/site/source/documentation/latest/persistent-volume.md Wed Sep 30 07:05:01 2015 @@ -2,7 +2,7 @@ layout: documentation --- -# Persistent Volume +# Persistent Volumes Mesos provides a mechanism to create a persistent volume from disk resources. When launching a task, you can create a volume that exists outside Modified: mesos/site/source/documentation/latest/reconciliation.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/reconciliation.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/reconciliation.md (original) +++ mesos/site/source/documentation/latest/reconciliation.md Wed Sep 30 07:05:01 2015 @@ -2,17 +2,17 @@ There's no getting around it, **frameworks on Mesos are distributed systems**. -**Distributed systems must deal with failures**, and partitions (the two are +**Distributed systems must deal with failures** and partitions (the two are indistinguishable from a system's perspective). Concretely, what does this mean for frameworks? Mesos uses an actor-like -**message passing programming model, in which messages are delivered -at-most-once**. (Exceptions to this include task status updates, most of +**message passing** programming model, in which messages are delivered +**at-most-once**. (Exceptions to this include task status updates, most of which are delivered at-least-once through the use of acknowledgements). **The messages passed between the master and the framework are therefore susceptible to be dropped, in the presence of failures**. -When these non-reliable messages are dropped, inconsistent state can arise +When these unreliable messages are dropped, inconsistent state can arise between the framework and Mesos. As a simple example, consider a launch task request sent by a framework. Modified: mesos/site/source/documentation/latest/reporting-a-bug.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/reporting-a-bug.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/reporting-a-bug.md (original) +++ mesos/site/source/documentation/latest/reporting-a-bug.md Wed Sep 30 07:05:01 2015 @@ -8,4 +8,4 @@ When reporting an issue (e.g. bug, impro * We track all issues via Apache's hosted [JIRA issue tracker](https://issues.apache.org/jira/browse/MESOS), so if you don't already have an account you'll need to register for one. It's quick and easy. * A JIRA should be created for every task, feature, bug-fix, etc. This makes it easy to track progress. -* If you are planning to work on the ticket, please assign the JIRA issue to yourself before you start working on it. This helps to avoid duplication of work, and alerts anyone who is following that issue. It is highly encouraged to start a discussion with others in the community about how to solve the issue. \ No newline at end of file +* If you are planning to work on the ticket, please assign the JIRA issue to yourself before you start working on it. This helps to avoid duplication of work and alerts anyone who is following that issue. It is highly encouraged to start a discussion with others in the community about how to solve the issue. The [contribution guidelines](/documentation/latest/submitting-a-patch/) discuss how to contribute to Mesos development in more detail. \ No newline at end of file Modified: mesos/site/source/documentation/latest/scheduler_http_api.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/scheduler_http_api.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/scheduler_http_api.md (original) +++ mesos/site/source/documentation/latest/scheduler_http_api.md Wed Sep 30 07:05:01 2015 @@ -471,7 +471,7 @@ In the case of a network partition, the ## Master detection -Mesos has a high-availability mode that uses multiple Mesos masters; one active master (/documentation/latest/called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the [documentation](high-availability/). +Mesos has a high-availability mode that uses multiple Mesos masters; one active master (/documentation/latest/called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the [documentation](/documentation/latest/high-availability/). Schedulers are expected to make HTTP requests to the leading master. If requests are made to a non-leading master a âHTTP 307 Temporary Redirectâ will be received with the âLocationâ header pointing to the leading master. Modified: mesos/site/source/documentation/latest/submitting-a-patch.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/submitting-a-patch.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/submitting-a-patch.md (original) +++ mesos/site/source/documentation/latest/submitting-a-patch.md Wed Sep 30 07:05:01 2015 @@ -8,23 +8,31 @@ layout: documentation * We use [Apache Review Board](https://reviews.apache.org) for code reviews. If you don't already have an account, you'll need to create one (it's separate from your Apache JIRA account). * A code review request should be created for every JIRA that involves a change to the codebase. -### Before you submit +### Before you start writing code 1. Check out the code from the Apache repository via Git. Instructions are on the [Getting Started](http://mesos.apache.org/gettingstarted/) page. -2. Join the dev, issues, reviews and builds mailing lists by sending emails to [email protected], [email protected], [email protected] and [email protected] respectively. +2. Join the dev, issues, reviews and builds mailing lists by sending emails to [[email protected]](mailto:[email protected]), [[email protected]](mailto:[email protected]), [[email protected]](mailto:[email protected]) and [[email protected]](mailto:[email protected]) respectively. -3. Find a JIRA that is currently unassigned that you want to work on at [JIRA issue tracker](https://issues.apache.org/jira/browse/MESOS), or create your own (you'll need a JIRA account for this, see below)! +3. Find a JIRA issue that is currently unassigned that you want to work on at [JIRA issue tracker](https://issues.apache.org/jira/browse/MESOS), or create your own (you'll need a JIRA account for this, see below)! 1. This could be a JIRA representing a bug (possibly a bug that you encountered and reported, e.g. when trying to build) or a new feature. + 2. Prefer working on issues marked as "[Accepted](https://issues.apache.org/jira/browse/MESOS-1?jql=project%20%3D%20MESOS%20AND%20status%20%3D%20Accepted)", rather than merely "Open". If an issue has been accepted, it means at least one Mesos developer thought that the ideas proposed in the issue are worth pursuing further. + 3. Issues marked with the "[newbie](https://issues.apache.org/jira/browse/MESOS-1?jql=project%20%3D%20MESOS%20AND%20status%20%3D%20Accepted%20AND%20labels%20%3D%20newbie)" label can be good candidates for "starter" projects. 4. Assign the JIRA to yourself. To do this, you will need: 1. An Apache JIRA user account (sign up for one [here](https://issues.apache.org/jira/secure/Signup!default.jspa)). - 2. You need to be added to the list of Mesos "contributors" by a Mesos committer (send email to [email protected]) in order to be assigned (or to assign yourself) to a JIRA issue. + 2. You need to be added to the list of Mesos "contributors" by a Mesos committer (send email to [[email protected]](mailto:[email protected])) in order to be assigned (or to assign yourself) to a JIRA issue. -5. Formulate a plan for resolving the issue, propose your plan via comments in the JIRA. +5. Formulate a plan for resolving the issue. Propose your plan via comments in the JIRA. -6. Create one or more test cases to exercise the bug or the feature (the Mesos team uses [test-driven development](http://en.wikipedia.org/wiki/Test-driven_development)), before you start coding, make sure these test cases all fail. +6. Find a **shepherd** to collaborate on your patch. A shepherd is a Mesos committer that will work with you to give you feedback on your proposed design, and to eventually commit your change into the Mesos source tree. + 1. To find a shepherd, email the dev mailing list (include a link to your JIRA issue). You can also try asking by adding a comment to your JIRA issue. + 2. You can also find a shepherd by asking the developers on IRC (/documentation/latest/in the [mesos channel](irc://irc.freenode.net/mesos) on [Freenode](https://freenode.net)). You can find the current list of committers [here](committers/): a developer that has previously worked on the component you are modifying might be a good candidate shepherd. -7. Make your changes to the code (using whatever IDE/editor you choose) to actually fix the bug or implement the feature. +### Create your patch +1. Create one or more test cases to exercise the bug or the feature (the Mesos team uses [test-driven development](http://en.wikipedia.org/wiki/Test-driven_development)). Before you start coding, make sure these test cases all fail. + 1. The [testing patterns](/documentation/latest/mesos-testing-patterns/) page has some suggestions for writing test cases. + +2. Make your changes to the code (using whatever IDE/editor you choose) to actually fix the bug or implement the feature. 1. Before beginning, please read the [Mesos C++ Style Guide](/documentation/latest/mesos-c++-style-guide/). It is recommended to use the git pre-commit hook (`support/hooks/pre-commit`) to automatically check for style errors. See the hook script for instructions to enable it. 2. Most of your changes will probably be to files inside of `BASE_MESOS_DIR` 3. From inside of the root Mesos directory: `./bootstrap` (Only required if building from git repository). @@ -34,10 +42,10 @@ layout: documentation 3. `make` 4. Now all of the files generated by the build process will be contained in the build directory you created, instead of being spread throughout the src directory, which is a bit messier. This is both cleaner, and makes it easy to clean up if you want to get rid of the files generated by `configure` and `make`. I.e. You can reset your build process without risking changes you made in the src directory, by simply deleting the build directory, and creating a new one. -8. Make sure all of your test cases now pass. - 1. `make check` +3. Make sure that all of the unit tests pass, including the new test cases you have added: `make check`. + 1. To execute a single unit test (helpful when trying to debug a test case failure), use something like: `make check GTEST_FILTER="HTTPTest.Delete"`. -9. Make sure to pull in any changes that have been committed to master branch. Using Git, do this via something like: +4. Make sure to pull in any changes that have been committed to master branch. Using Git, do this via something like: 1. `git checkout master` 2. `git pull` 3. `git checkout my_branch` @@ -45,25 +53,28 @@ layout: documentation ### Submit your patch 1. You're ready to submit your patch for review! - 1. Log in or create an account at [Apache Review Board](http://reviews.apache.org) + 1. Log in or create an account at [Apache Review Board](http://reviews.apache.org). 2. The easiest (and recommended) way to submit reviews is through `post-reviews.py` a wrapper around post-review. - 3. First, install RBTools. [See Instructions](https://www.reviewboard.org/docs/rbtools/dev/) - 4. Configure post-review. The easiest method is to symlink to the sample config: ln -s support/reviewboardrc .reviewboardrc - 5. Log into Review Board from the command line: run `rbt status` + 3. First, install RBTools. [See Instructions](https://www.reviewboard.org/docs/rbtools/dev/). + 4. Configure post-review. The easiest method is to symlink to the sample config: `ln -s support/reviewboardrc .reviewboardrc`. + 5. Log into Review Board from the command line: run `rbt status`. 6. From your local branch run `support/post-reviews.py`. 7. Note that `post-reviews.py` creates a new review for every commit on your branch that is different from the `master`. - 8. Be sure to add your JIRA issue id (e.g. MESOS-01) to the field labeled "Bugs" (this will automatically link) - 9. Under "Description" in addition to details about your changes, include a description of any documentation pages that need to be added, or are affected by your changes (e.g. did you change or add any configuration options/flags? Did you add a new binary?) + 8. Be sure to add your JIRA issue id (e.g. MESOS-1) to the field labeled "Bugs" (this will automatically link). + 9. Add your shepherd under the "People" field, in the "Reviewers" section. You should also include other Mesos community members who have contributed to the discussion of your proposed change. + 10. Under "Description" in addition to details about your changes, include a description of any documentation pages that need to be added, or are affected by your changes (e.g. did you change or add any configuration options/flags? Did you add a new binary?) + 11. Under "Testing Done", explain what new tests you have created, what tests were modified, and what procedures you went through to test your changes. 2. Wait for a code review from another Mesos developer via Review Board, address their feedback and upload updated patches until you receive a "Ship It" from a Mesos committer. - 1. When addressing feedback, adjust your existing commit(s) instead of creating new commits, otherwise `post-reviews.py` will create a new review (`git rebase -i` is your friend). - 2. Review Board comments should be used for code-specific discussions, and JIRA comments for bigger-picture design discussions. - 3. Always respond to each RB comment that you address directly (i.e. each comment can be responded to directly) with either "Done." or a comment explaining how you addressed it. - 4. If an issue has been raised in the review, please resolve the issue as "Fixed" or "Dropped". If "Dropped" please add a comment explaining the reason. Also, if your fix warrants a comment (e.g., fixed differently than suggested by the reviewer) please add a comment. + 1. If you don't receive any feedback, contact your shepherd to remind them. While the committers try their best to provide prompt feedback on proposed changes, they are busy and sometimes a patch gets overlooked. + 2. When addressing feedback, adjust your existing commit(s) instead of creating new commits, otherwise `post-reviews.py` will create a new review (`git rebase -i` is your friend). + 3. Review Board comments should be used for code-specific discussions, and JIRA comments for bigger-picture design discussions. + 4. Always respond to each RB comment that you address directly (i.e. each comment can be responded to directly) with either "Done." or a comment explaining how you addressed it. + 5. If an issue has been raised in the review, please resolve the issue as "Fixed" or "Dropped". If "Dropped" please add a comment explaining the reason. Also, if your fix warrants a comment (e.g., fixed differently than suggested by the reviewer) please add a comment. 3. After consensus is reached on your JIRA/patch, you're review request will receive a "Ship It!" from a committer, and then a committer will commit your patch to the git repository. Congratulations and thanks for participating in our community! 4. The last step is to ensure that the necessary documentation gets created or updated so the whole world knows about your new feature or bug fix. ## Style Guides -* For patches to the core, we ask that you follow the [Mesos C++ Style Guide](/documentation/latest/mesos-c++-style-guide/) +* For patches to the core, we ask that you follow the [Mesos C++ Style Guide](/documentation/latest/mesos-c++-style-guide/). Modified: mesos/site/source/documentation/latest/tools.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/tools.md?rev=1705971&r1=1705970&r2=1705971&view=diff ============================================================================== --- mesos/site/source/documentation/latest/tools.md (original) +++ mesos/site/source/documentation/latest/tools.md Wed Sep 30 07:05:01 2015 @@ -20,6 +20,7 @@ These tools make it easy to set up and r If you want to hack on Mesos or write a new framework, these tools will help. +* [clang-format](/documentation/latest/clang-format/) to automatically apply some of the style rules dictated by the [Mesos C++ Style Guide](/documentation/latest/mesos-c++-style-guide/). * [Go Bindings and Examples](https://github.com/mesosphere/mesos-go) Write a Mesos framework in Go! Comes with an example scheduler and executor. * [Mesos Framework giter8 Template](https://github.com/mesosphere/scala-sbt-mesos-framework.g8) This is a giter8 template. The result of applying this template is a bare-bones Apache Mesos framework in Scala using SBT for builds and Vagrant for testing on a singleton cluster. * [Scala Hello World](https://gist.github.com/guenter/7471695) A simple Mesos "Hello World": downloads and starts a web server on every node in the cluster.
