Author: dlester
Date: Wed Sep 30 07:05:01 2015
New Revision: 1705971
URL: http://svn.apache.org/viewvc?rev=1705971&view=rev
Log:
Update documentation on website.
Modified:
mesos/site/publish/documentation/allocation-module/index.html
mesos/site/publish/documentation/app-framework-development-guide/index.html
mesos/site/publish/documentation/attributes-resources/index.html
mesos/site/publish/documentation/deploy-scripts/index.html
mesos/site/publish/documentation/fetcher-cache-internals/index.html
mesos/site/publish/documentation/fetcher/index.html
mesos/site/publish/documentation/index.html
mesos/site/publish/documentation/latest/allocation-module/index.html
mesos/site/publish/documentation/latest/app-framework-development-guide/index.html
mesos/site/publish/documentation/latest/attributes-resources/index.html
mesos/site/publish/documentation/latest/deploy-scripts/index.html
mesos/site/publish/documentation/latest/fetcher-cache-internals/index.html
mesos/site/publish/documentation/latest/fetcher/index.html
mesos/site/publish/documentation/latest/index.html
mesos/site/publish/documentation/latest/mesos-c++-style-guide/index.html
mesos/site/publish/documentation/latest/oversubscription/index.html
mesos/site/publish/documentation/latest/persistent-volume/index.html
mesos/site/publish/documentation/latest/reconciliation/index.html
mesos/site/publish/documentation/latest/reporting-a-bug/index.html
mesos/site/publish/documentation/latest/scheduler_http_api/index.html
mesos/site/publish/documentation/latest/submitting-a-patch/index.html
mesos/site/publish/documentation/latest/tools/index.html
mesos/site/publish/documentation/mesos-c++-style-guide/index.html
mesos/site/publish/documentation/oversubscription/index.html
mesos/site/publish/documentation/persistent-volume/index.html
mesos/site/publish/documentation/reconciliation/index.html
mesos/site/publish/documentation/reporting-a-bug/index.html
mesos/site/publish/documentation/scheduler_http_api/index.html
mesos/site/publish/documentation/submitting-a-patch/index.html
mesos/site/publish/documentation/tools/index.html
mesos/site/source/documentation/latest.html.md
mesos/site/source/documentation/latest/allocation-module.md
mesos/site/source/documentation/latest/app-framework-development-guide.md
mesos/site/source/documentation/latest/attributes-resources.md
mesos/site/source/documentation/latest/deploy-scripts.md
mesos/site/source/documentation/latest/fetcher-cache-internals.md
mesos/site/source/documentation/latest/fetcher.md
mesos/site/source/documentation/latest/mesos-c++-style-guide.md
mesos/site/source/documentation/latest/oversubscription.md
mesos/site/source/documentation/latest/persistent-volume.md
mesos/site/source/documentation/latest/reconciliation.md
mesos/site/source/documentation/latest/reporting-a-bug.md
mesos/site/source/documentation/latest/scheduler_http_api.md
mesos/site/source/documentation/latest/submitting-a-patch.md
mesos/site/source/documentation/latest/tools.md
Modified: mesos/site/publish/documentation/allocation-module/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/allocation-module/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/allocation-module/index.html (original)
+++ mesos/site/publish/documentation/allocation-module/index.html Wed Sep 30
07:05:01 2015
@@ -81,14 +81,14 @@
<p>See our <a href="/community/">community</a> page for more
details.</p>
</div>
<div class="col-md-8">
- <h1>Mesos Allocation Module</h1>
+ <h1>Mesos Allocation Modules</h1>
-<p>The logic that the Mesos master uses to determine which frameworks to make
resource offers to is encapsulated in the Master’s <em>allocator
module</em>. The allocator is a pluggable component that organizations can use
to implement their own sharing policy, e.g. fair-sharing, priority, etc., or
tune the default hierarchical Dominant Resource Fairness algorithm (see <a
href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf";>the DRF
paper</a>).</p>
+<p>The logic that the Mesos master uses to determine which frameworks to make
resource offers to is encapsulated in the master’s <em>allocator
module</em>. The allocator is a pluggable component that organizations can use
to implement their own sharing policy, e.g. fair-sharing, priority, etc., or
tune the default hierarchical Dominant Resource Fairness algorithm (see <a
href="https://www.cs.berkeley.edu/~alig/papers/drf.pdf";>the DRF paper</a>).</p>
<p>To use a custom allocator in Mesos, one must:</p>
<ul>
-<li><p><a href="#writing-a-custom-allocator">Implement</a> an
<code>Allocator</code> interface as defined in
<code>mesos/master/allocator.hpp</code>,</p></li>
+<li><p><a href="#writing-a-custom-allocator">Implement</a> the
<code>Allocator</code> interface as defined in
<code>mesos/master/allocator.hpp</code>,</p></li>
<li><p><a href="#wiring-up-a-custom-allocator">Wrap</a> the allocator
implementation in a module and load it in the Mesos master.</p></li>
</ul>
@@ -101,7 +101,7 @@
<p>Additionally, the built-in hierarchical allocator can be extended without
the need to reimplement the entirety of the allocation logic. This is possible
through the use of the <code>Sorter</code> abstraction. Sorters define the
order in which hierarchy layers (e.g. roles or frameworks) should be offered
resources by taking “client” objects and some information about
those clients and returning an ordered list of clients.</p>
-<p>Sorters are implemented in C++ and inherit the <code>Sorter</code> class
defined in <code>$MESOS_HOME/src/master/allocator/sorter/sorter.hpp</code>. The
default sorter is <code>DRFSorter</code>, which implements fair sharing and can
be found in
<code>$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp</code>. This
sorter is capable of expressing priorities by specifying weights in
<code>Sorter::add()</code>. Each client’s share is divided by its weight.
For example, a role that has a weight of <code>2</code> will be offered twice
as many resources as a role with weight <code>1</code>.</p>
+<p>Sorters are implemented in C++ and inherit the <code>Sorter</code> class
defined in <code>$MESOS_HOME/src/master/allocator/sorter/sorter.hpp</code>. The
default sorter is <code>DRFSorter</code>, which implements fair sharing and can
be found in
<code>$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp</code>. This
sorter is capable of expressing priorities by specifying weights in
<code>Sorter::add()</code>. Each client’s share is divided by its weight.
For example, a role that has a weight of 2 will be offered twice as many
resources as a role with weight 1.</p>
<h2>Wiring up a custom allocator</h2>
Modified:
mesos/site/publish/documentation/app-framework-development-guide/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/app-framework-development-guide/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/app-framework-development-guide/index.html
(original)
+++ mesos/site/publish/documentation/app-framework-development-guide/index.html
Wed Sep 30 07:05:01 2015
@@ -283,17 +283,17 @@ virtual void error(ExecutorDriver* drive
<h2>Install your Framework</h2>
-<p>You need to put your framework somewhere that all slaves on the cluster can
get it from. If you are running HDFS, you can put your executor into HDFS.
Then, you tell Mesos where it is via the <code>ExecutorInfo</code> parameter of
<code>MesosSchedulerDriver</code>’s constructor (e.g. see
src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a
a Protocol Buffer Message class (defined in
<code>include/mesos/mesos.proto</code>), and you set its URI field to something
like “HDFS://path/to/executor/”. Also, you can pass the
<code>frameworks_home</code> configuration option (defaults to:
<code>MESOS_HOME/frameworks</code>) to your <code>mesos-slave</code> daemons
when you launch them to specify where all of your framework executors are
stored (e.g. on an NFS mount that is available to all slaves), then set
<code>ExecutorInfo</code> to be a relative path, and the slave will prepend the
value of frameworks_home to the relative path provided.</p>
+<p>You need to put your framework somewhere that all slaves on the cluster can
get it from. If you are running HDFS, you can put your executor into HDFS.
Then, you tell Mesos where it is via the <code>ExecutorInfo</code> parameter of
<code>MesosSchedulerDriver</code>’s constructor (e.g. see
src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a
Protocol Buffer Message class (defined in
<code>include/mesos/mesos.proto</code>), and you set its URI field to something
like “HDFS://path/to/executor/”. Also, you can pass the
<code>frameworks_home</code> configuration option (defaults to:
<code>MESOS_HOME/frameworks</code>) to your <code>mesos-slave</code> daemons
when you launch them to specify where all of your framework executors are
stored (e.g. on an NFS mount that is available to all slaves), then set
<code>ExecutorInfo</code> to be a relative path, and the slave will prepend the
value of frameworks_home to the relative path provided.</p>
<p>Once you are sure that your executors are available to the mesos-slaves,
you should be able to run your scheduler, which will register with the Mesos
master, and start receiving resource offers!</p>
<h2>Labels</h2>
-<p><code>Labels</code> can be found in the <code>TaskInfo</code>,
<code>DiscoveryInfo</code> and <code>TaskStatus</code>s and
-let’s framework and module writers use Labels to tag and pass
unstructured
-information around Mesos. Labels are free-form key-value pairs supplied by the
-framework scheduler or label decorator hooks. Below is the protobuf definitions
-of labels:</p>
+<p><code>Labels</code> can be found in the <code>FrameworkInfo</code>,
<code>TaskInfo</code>, <code>DiscoveryInfo</code> and
+<code>TaskStatus</code> messages; framework and module writers can use Labels
to tag and
+pass unstructured information around Mesos. Labels are free-form key-value
pairs
+supplied by the framework scheduler or label decorator hooks. Below is the
+protobuf definitions of labels:</p>
<pre><code class="{.proto}"> optional Labels labels = 11;
</code></pre>
Modified: mesos/site/publish/documentation/attributes-resources/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/attributes-resources/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/attributes-resources/index.html (original)
+++ mesos/site/publish/documentation/attributes-resources/index.html Wed Sep 30
07:05:01 2015
@@ -108,7 +108,7 @@ text : [a-zA-Z0-9_/.-]
<h2>Attributes</h2>
-<p>Attributes are key value pairs (where value is optional) that Mesos passes
along when it sends offers to frameworks. An attribute value supports 3
different <em>types</em>: scalar, range or text.</p>
+<p>Attributes are key-value pairs (where value is optional) that Mesos passes
along when it sends offers to frameworks. An attribute value supports 3
different <em>types</em>: scalar, range or text.</p>
<pre><code>attributes : attribute ( ";" attribute )*
Modified: mesos/site/publish/documentation/deploy-scripts/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/deploy-scripts/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/deploy-scripts/index.html (original)
+++ mesos/site/publish/documentation/deploy-scripts/index.html Wed Sep 30
07:05:01 2015
@@ -81,13 +81,21 @@
<p>See our <a href="/community/">community</a> page for more
details.</p>
</div>
<div class="col-md-8">
- <h1>Deploy Scripts</h1>
+ <h1>Deployment Scripts</h1>
-<p>Mesos includes a set of scripts in <code>[install-prefix]/sbin</code> that
can be used to deploy it on a cluster. To use these scripts, you need to create
two configuration files:
<code>[install-prefix]/var/mesos/deploy/masters</code>, which should list the
hostname(s) of the node(s) you want to be your masters (one per line), and
<code>[install-prefix]/var/mesos/deploy/slaves</code>, which should contain a
list of hostnames for your slaves (one per line). You can then start a cluster
with <code>[install-prefix]/sbin/mesos-start-cluster.sh</code> and stop it with
<code>[install-prefix]/sbin/mesos-stop-cluster.sh</code>.</p>
+<p>Mesos includes a set of scripts in <code>[install-prefix]/sbin</code> that
can be used to deploy it on a cluster. To use these scripts, you need to create
two configuration files:</p>
-<p>It is also possible to set environment variables, ulimits, etc that will
affect the master and slave. by editing
<code>[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh</code>. One
particularly useful setting is <code>LIBPROCESS_IP</code>, which tells the
master and slave binaries which IP address to bind to; in some installations,
the default interface that the hostname resolves to is not the machine’s
external IP address, so you can set the right IP through this variable. Besides
the common environment variables of master and slave configured in
<code>[install-prefix/var/mesos/deploy/mesos-deploy-env.sh</code>, it is also
possible to set master specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-master-env.sh</code>, slave
specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-slave-env.sh</code>.</p>
+<ol>
+<li><code>[install-prefix]/var/mesos/deploy/masters</code>, which should list
the hostname(s) of the node(s) you want to be your masters (one per line),
and</li>
+<li><code>[install-prefix]/var/mesos/deploy/slaves</code>, which should
contain a list of hostnames for your slaves (one per line).</li>
+</ol>
-<p>Finally, the deploy scripts do not use ZooKeeper by default. If you want to
use ZooKeeper (/documentation/latest/for multiple masters). Please see the <a
href="high-availability/">High Availability</a> documentation for details.</p>
+
+<p>You can then start a cluster with
<code>[install-prefix]/sbin/mesos-start-cluster.sh</code> and stop it with
<code>[install-prefix]/sbin/mesos-stop-cluster.sh</code>.</p>
+
+<p>It is also possible to set environment variables, ulimits, etc that will
affect the master and slave by editing
<code>[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh</code>. One
particularly useful setting is <code>LIBPROCESS_IP</code>, which tells the
master and slave binaries which IP address to bind to; in some installations,
the default interface that the hostname resolves to is not the machine’s
external IP address, so you can set the right IP through this variable. Besides
the common environment variables of master and slave configured in
<code>[install-prefix/var/mesos/deploy/mesos-deploy-env.sh</code>, it is also
possible to set master specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-master-env.sh</code>, slave
specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-slave-env.sh</code>.</p>
+
+<p>Finally, the deploy scripts do not use ZooKeeper by default. If you want to
configure Mesos to use ZooKeeper to coordinate multiple master nodes, please
see the <a href="/documentation/latest/high-availability/">High
Availability</a> documentation for details.</p>
<h2>Notes</h2>
Modified: mesos/site/publish/documentation/fetcher-cache-internals/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/fetcher-cache-internals/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/fetcher-cache-internals/index.html
(original)
+++ mesos/site/publish/documentation/fetcher-cache-internals/index.html Wed Sep
30 07:05:01 2015
@@ -83,7 +83,7 @@
<div class="col-md-8">
<h1>Mesos Fetcher Cache Internals</h1>
-<p>It assumed that readers of this document are well familiar with the
contents of the overview and user guide of the Mesos fetcher in
“fetcher.md”. The present document makes direct references to
notions defined in the former.</p>
+<p>It assumed that readers of this document are familiar with the contents of
the <a href="/documentation/latest/fetcher/">Mesos fetcher user guide</a>. The
present document makes direct references to notions defined in the former.</p>
<h2>Design goals for the initial fetcher cache prototype:</h2>
@@ -97,9 +97,13 @@
</ol>
-<p>For future releases, we foresee additional features:
-1. Automatic refreshing of cache content when a URI’s content has
changed.
-2. Prefetching URIs for subsequent tasks. Prefetching can run in parallel with
task execution.</p>
+<p>For future releases, we foresee additional features:</p>
+
+<ol>
+<li>Automatic refreshing of cache content when a URI’s content has
changed.</li>
+<li>Prefetching URIs for subsequent tasks. Prefetching can run in parallel
with task execution.</li>
+</ol>
+
<h2>How the fetcher cache works</h2>
@@ -138,7 +142,7 @@
<p>The fetcher process uses a private instance of class Cache to represent
what URIs are cached, where the respective cache files are, what stage of
processing they are in, and so on.</p>
-<p>The main data structure to hold all this information is a hashmap from
URI/user combinations to Cache::Entry objects, which each contain information
about an individual cache file on disk. These objects are referenced by
shared_ptr, because they can be addressed by multiple callbacks on behalf of
concurrent fetch attempts while also being held in the hashmap.</p>
+<p>The main data structure to hold all this information is a hashmap from
URI/user combinations to <code>Cache::Entry</code> objects, which each contain
information about an individual cache file on disk. These objects are
referenced by shared_ptr, because they can be addressed by multiple callbacks
on behalf of concurrent fetch attempts while also being held in the hashmap.</p>
<p>A cache entry corresponds directly to a cache file on disk throughout the
entire life time of the latter, including before and after its existence. It
holds all pertinent state to inform about the phase and results of fetching the
corresponding URI.</p>
@@ -150,10 +154,14 @@
<p>The two blue states are essentially the same: no cache file exists. The two
green disk states on the right are also the same.</p>
-<p>The figure only depicts what happens from the point of view of one isolated
fetch run. Any given cache entry can be referenced simultaniously by another
concurrent fetch run. It must not be evicted as long as it is referenced by any
fetching activity. We implement this by reference counting. Every cache entry
has a reference count field that gets incremented at the beginning of its use
by a fetch run and decremented at its end. The latter must happen no matter
whether the run has been successful or whether there has been an error.
Increments happen when:
-- A new cache entry is created. It is immediately referenced.
-- An existing cache entry’s file download is going to be waited for.
-- An existing cache entry has a resident cache file that is going to be
retrieved.</p>
+<p>The figure only depicts what happens from the point of view of one isolated
fetch run. Any given cache entry can be referenced simultaniously by another
concurrent fetch run. It must not be evicted as long as it is referenced by any
fetching activity. We implement this by reference counting. Every cache entry
has a reference count field that gets incremented at the beginning of its use
by a fetch run and decremented at its end. The latter must happen no matter
whether the run has been successful or whether there has been an error.
Increments happen when:</p>
+
+<ul>
+<li>A new cache entry is created. It is immediately referenced.</li>
+<li>An existing cache entry’s file download is going to be waited
for.</li>
+<li>An existing cache entry has a resident cache file that is going to be
retrieved.</li>
+</ul>
+
<p>Every increment is recorded in a list. At the very end of the fetch
procedure, no matter what its outcome is, each entry in the list gets its
reference count decremented.</p>
@@ -165,17 +173,25 @@
<p><img src="images/fetch_flow.jpg?raw=true" alt="Determining Fetcher Actions"
/></p>
-<p>After going through this procedure for each URI, the fetcher process
assembles the gathered list of per-URI actions into a JSON object
(FetcherInfo), which is passed to the mesos-fetcher program in an environment
variable. The possible fetch actions for a URI are shown at the bottom of the
flow chart. After they are determined, the fetcher process invokes
mesos-fetcher.</p>
+<p>After going through this procedure for each URI, the fetcher process
assembles the gathered list of per-URI actions into a JSON object
(<code>FetcherInfo</code>), which is passed to the mesos-fetcher program in an
environment variable. The possible fetch actions for a URI are shown at the
bottom of the flow chart. After they are determined, the fetcher process
invokes mesos-fetcher.</p>
<p>The implementation is oriented at this control flow but its code structure
cannot match it directly, because some of these branches must span multiple
libprocess continuations. There are two layers of futures, one for each of
these phases.</p>
<ol>
-<li>Before making fetcher cache items.</li>
-<li>a) Wait for concurrent downloads for pre-existing cache entries</li>
-<li><p>b) Wait for size fetching combined and then space reservation for new
cache entries.</p></li>
-<li><p>After making fetcher cache items and running mesos-fetcher.</p></li>
+<li> Before making fetcher cache items,
+
+<ol type="a">
+<li>Wait for concurrent downloads for pre-existing cache entries.</li>
+<li>Wait for size fetching combined and then space reservation for new cache
entries.</li>
+</ol>
+</li>
+<li> After making fetcher cache items and running mesos-fetcher,
+
+<ol type="a">
<li>Complete new cache items with success/failure, which as an important
side-effect informs concurrent fetch runs' futures in phase 1/a.</li>
</ol>
+</li>
+</ol>
<p>The futures for phase 1 are not shared outside one fetch run. They
exclusively guard asynchronous operations for the same fetch run. Their type
parameter does not really matter. But each needs to correspond to one URI and
eventual fetch item somehow. Multiple variants have been proposed for this. The
complexity remains about the same.</p>
@@ -192,9 +208,13 @@
<p>The resources named “A” and “B” have been fetched
with caching into sandbox 1 and 2 below. In the course of this, two cache
entries have been created and two files have been downloaded into the cache and
named “1” and “2”. (Cache file names have unique names
that comprise serial numbers.)</p>
-<p>The next figure illustrates the state after fetching a different cached URI
into sandbox 3, which in this case requires evicting a cache-resident file and
its entry. Cache eviction removes cache entries in the order of the least
recently used cache entries. Steps if “A” was fetched before
“B”:
-1. Remove the cache entry for “A” from the fetcher process' cache
entry table. Its faded depiction is supposed to indicate this. This immediately
makes it appear as if the URI has never been cached, even though the cache file
is still around.
-2. Proceed with fetching “C”. This creates a new cache file, which
has a different unique name. (The fetcher process remembers in its cache entry
which file name belongs to which URI.)</p>
+<p>The next figure illustrates the state after fetching a different cached URI
into sandbox 3, which in this case requires evicting a cache-resident file and
its entry. Cache eviction removes cache entries in the order of the least
recently used cache entries. Steps if “A” was fetched before
“B”:</p>
+
+<ol>
+<li>Remove the cache entry for “A” from the fetcher process' cache
entry table. Its faded depiction is supposed to indicate this. This immediately
makes it appear as if the URI has never been cached, even though the cache file
is still around.</li>
+<li>Proceed with fetching “C”. This creates a new cache file,
which has a different unique name. (The fetcher process remembers in its cache
entry which file name belongs to which URI.)</li>
+</ol>
+
<p><img src="images/fetch_evict2.jpg?raw=true" alt="After eviction" /></p>
Modified: mesos/site/publish/documentation/fetcher/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/fetcher/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/fetcher/index.html (original)
+++ mesos/site/publish/documentation/fetcher/index.html Wed Sep 30 07:05:01 2015
@@ -83,8 +83,7 @@
<div class="col-md-8">
<h1>Mesos Fetcher</h1>
-<p>Experimental support for the Mesos fetcher <em>cache</em> is introduced in
-Mesos 0.23.0.</p>
+<p>Mesos 0.23.0 introduced experimental support for the Mesos <em>fetcher
cache</em>.</p>
<p>In this context we loosely regard the term “downloading” as to
include copying
from local file systems.</p>
@@ -94,7 +93,7 @@ from local file systems.</p>
<p>The Mesos fetcher is a mechanism to download resources into the sandbox
directory of a task in preparation of running the task. As part of a TaskInfo
message, the framework ordering the task’s execution provides a list of
-CommandInfo::URI protobuf values, which becomes the input to the Mesos
fetcher.</p>
+<code>CommandInfo::URI</code> protobuf values, which becomes the input to the
Mesos fetcher.</p>
<p>By default, each requested URI is downloaded directly into the sandbox
directory
and repeated requests for the same URI leads to downloading another copy of the
@@ -109,7 +108,7 @@ coordinates all fetch actions. Every sla
fetcher instance that is used by every kind of containerizer (except the
external containerizer variant, which is responsible for its own approach to
fetching).</p></li>
-<li><p>The external program “mesos-fetcher” that is invoked by the
former. It
+<li><p>The external program <code>mesos-fetcher</code> that is invoked by the
former. It
performs all network and disk operations except file deletions and file size
queries for cache-internal bookkeeping. It is run as an external OS process in
order to shield the slave process from I/O-related hazards. It takes
@@ -120,14 +119,14 @@ detailed fetch action descriptions.</p><
<h2>The fetch procedure</h2>
-<p>Frameworks launch tasks by calling the scheduler driver method
launchTasks(),
-passing CommandInfo protobuf structures as arguments. This type of structure
+<p>Frameworks launch tasks by calling the scheduler driver method
<code>launchTasks()</code>,
+passing <code>CommandInfo</code> protobuf structures as arguments. This type
of structure
specifies (among other things) a command and a list of URIs that need to be
“fetched” into the sandbox directory on the the slave node as a
precondition for
task execution. Hence, when the slave receives a request go launch a task, it
calls upon its fetcher, first, to provision the specified resources into the
sandbox directory. If fetching fails, the task is not started and the reported
-task status is TASK_FAILED.</p>
+task status is <code>TASK_FAILED</code>.</p>
<p>All URIs requested for a given task are fetched sequentially in a single
invocation of mesos-fetcher. Here, avoiding download concurrency reduces the
@@ -138,7 +137,7 @@ active concurrently due to multiple task
<p>Before mesos-fetcher is started, the specific fetch actions to be performed
for
each URI are determined based on the following protobuf structure. (See
-“include/mesos/mesos.proto” for more details.)</p>
+<code>include/mesos/mesos.proto</code> for more details.)</p>
<pre><code>message CommandInfo {
message URI {
@@ -169,7 +168,7 @@ the affected task.</p>
<p>If a user name is specified either way, the fetcher first validates that it
is
in fact a valid user name on the slave. If it is not, fetching fails right
here.
Otherwise, the sandbox directory is assigned to the specified user as owner
-(using chown) at the end of the fetch procedure, before task execution
begins.</p>
+(using <code>chown</code>) at the end of the fetch procedure, before task
execution begins.</p>
<p>The user name in play has an important effect on caching. Caching is
managed on
a per-user base, i.e. the combination of user name and “uri”
uniquely
@@ -259,7 +258,7 @@ the URI.</p>
<ul>
<li>Local file sizes are probed with systems calls (that follow symbolic
links).</li>
<li>HTTP/HTTPS URIs are queried for the “content-length” field in
the header. This
-is performed by CURL. The reported asset size must be greater than zero or
+is performed by <code>curl</code>. The reported asset size must be greater
than zero or
the URI is deemed invalid.</li>
<li>FTP/FTPS is not supported at the time of writing.</li>
<li>Everything else is queried by the local HDFS client.</li>
@@ -355,6 +354,10 @@ running the present task, right after fe
</ul>
+<h2>Implementation Details</h2>
+
+<p>The <a href="/documentation/latest/fetcher-cache-internals/">Mesos Fetcher
Cache Internals</a> describes how the fetcher cache is implemented.</p>
+
</div>
</div>
Modified: mesos/site/publish/documentation/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/index.html (original)
+++ mesos/site/publish/documentation/index.html Wed Sep 30 07:05:01 2015
@@ -94,10 +94,12 @@
<h2>Running Mesos</h2>
<ul>
+<li><a href="/documentation/latest/getting-started/">Getting Started</a> for
basic instructions on compiling and installing Mesos.</li>
<li><a href="/documentation/latest/configuration/">Configuration</a> for
command-line arguments.</li>
<li><a href="/documentation/latest/mesos-containerizer/">Mesos
Containerizer</a> default containerizer, supports both Linux and POSIX
systems.</li>
<li><a href="/documentation/latest/docker-containerizer/">Docker
Containerizer</a> for launching a Docker image as a Task, or as an
Executor.</li>
<li><a href="/documentation/latest/external-containerizer/">External
Containerizer</a> for custom containerization implementations.</li>
+<li><a href="/documentation/latest/authentication/">Framework
Authentication</a></li>
<li><a href="/documentation/latest/authorization/">Framework
Authorization</a></li>
<li><a href="/documentation/latest/framework-rate-limiting/">Framework Rate
Limiting</a></li>
<li><a href="/documentation/latest/logging-and-debugging/">Logging and
Debugging</a> for viewing Mesos and framework logs.</li>
@@ -112,10 +114,21 @@
</ul>
+<h2>Advanced Features</h2>
+
+<ul>
+<li><a href="/documentation/attributes-resources/">Attributes and
Resources</a> for how to describe the slaves that comprise a cluster.</li>
+<li><a href="/documentation/latest/fetcher/">Fetcher Cache</a> for how to
configure the Mesos fetcher cache.</li>
+<li><a href="/documentation/latest/oversubscription/">Oversubscription</a> for
how to configure Mesos to take advantage of unused resources to launch
“best-effort” tasks.</li>
+<li><a href="/documentation/latest/persistent-volume/">Persistent Volume</a>
for how to allow tasks to access persistent storage resources.</li>
+<li><a href="/documentation/latest/reservation/">Reservation</a> for how to
configure Mesos to allow slaves to reserve resources.</li>
+</ul>
+
+
<h2>Running Mesos Frameworks</h2>
<ul>
-<li><a href="/documentation/latest/mesos-frameworks/">Mesos frameworks</a> for
a list of apps built on top of Mesos, and instructions on how to run them.</li>
+<li><a href="/documentation/latest/mesos-frameworks/">Mesos frameworks</a> for
a list of apps built on top of Mesos and instructions on how to run them.</li>
</ul>
@@ -124,16 +137,18 @@
<ul>
<li><a href="/documentation/latest/app-framework-development-guide/">Framework
Development Guide</a> describes how to build applications on top of Mesos.</li>
<li><a href="/documentation/latest/reconciliation/">Reconciliation</a> for
ensuring a framework’s state remains eventually consistent in the face of
failures.</li>
+<li><a href="/documentation/latest/scheduler_http_api/">Scheduler HTTP API</a>
describes the new HTTP API for communication between schedulers and the Mesos
master.</li>
<li><a href="/api/latest/java/">Javadoc</a> documents the Mesos Java API.</li>
<li><a href="/api/latest/c++/namespacemesos.html">Doxygen</a> documents the
Mesos C++ API.</li>
<li><a href="/documentation/latest/tools/">Developer Tools</a> for hacking on
Mesos or writing frameworks.</li>
</ul>
-<h2>Mesos Modules</h2>
+<h2>Extending Mesos</h2>
<ul>
<li><a href="/documentation/latest/modules/">Mesos Modules</a> for specifying
Mesos modules for master, slave and tests.</li>
+<li><a href="/documentation/latest/allocation-module/">Allocation Modules</a>
for how to write custom resource allocators.</li>
</ul>
@@ -141,7 +156,7 @@
<ul>
<li><a href="/documentation/latest/reporting-a-bug/">Reporting an Issue,
Improvement, or Feature</a> for getting started with JIRA.</li>
-<li><a href="/documentation/latest/submitting-a-patch/">Submitting a Patch</a>
for getting started with ReviewBoard, and our tooling around it.</li>
+<li><a href="/documentation/latest/submitting-a-patch/">Submitting a Patch</a>
for getting started with ReviewBoard and our tooling around it.</li>
<li><a href="/documentation/latest/mesos-testing-patterns/">Testing
Patterns</a> for tips and tricks used in Mesos tests.</li>
<li><a href="/documentation/latest/effective-code-reviewing/">Effective Code
Reviewing</a> guidelines, tips, and learnings for how to do effective code
reviews.</li>
<li><a
href="/documentation/latest/engineering-principles-and-practices/">Engineering
Principles and Practices</a> to serve as a shared set of project-level values
for the community.</li>
Modified: mesos/site/publish/documentation/latest/allocation-module/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/allocation-module/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/allocation-module/index.html
(original)
+++ mesos/site/publish/documentation/latest/allocation-module/index.html Wed
Sep 30 07:05:01 2015
@@ -81,14 +81,14 @@
<p>See our <a href="/community/">community</a> page for more
details.</p>
</div>
<div class="col-md-8">
- <h1>Mesos Allocation Module</h1>
+ <h1>Mesos Allocation Modules</h1>
-<p>The logic that the Mesos master uses to determine which frameworks to make
resource offers to is encapsulated in the Master’s <em>allocator
module</em>. The allocator is a pluggable component that organizations can use
to implement their own sharing policy, e.g. fair-sharing, priority, etc., or
tune the default hierarchical Dominant Resource Fairness algorithm (see <a
href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf";>the DRF
paper</a>).</p>
+<p>The logic that the Mesos master uses to determine which frameworks to make
resource offers to is encapsulated in the master’s <em>allocator
module</em>. The allocator is a pluggable component that organizations can use
to implement their own sharing policy, e.g. fair-sharing, priority, etc., or
tune the default hierarchical Dominant Resource Fairness algorithm (see <a
href="https://www.cs.berkeley.edu/~alig/papers/drf.pdf";>the DRF paper</a>).</p>
<p>To use a custom allocator in Mesos, one must:</p>
<ul>
-<li><p><a href="#writing-a-custom-allocator">Implement</a> an
<code>Allocator</code> interface as defined in
<code>mesos/master/allocator.hpp</code>,</p></li>
+<li><p><a href="#writing-a-custom-allocator">Implement</a> the
<code>Allocator</code> interface as defined in
<code>mesos/master/allocator.hpp</code>,</p></li>
<li><p><a href="#wiring-up-a-custom-allocator">Wrap</a> the allocator
implementation in a module and load it in the Mesos master.</p></li>
</ul>
@@ -101,7 +101,7 @@
<p>Additionally, the built-in hierarchical allocator can be extended without
the need to reimplement the entirety of the allocation logic. This is possible
through the use of the <code>Sorter</code> abstraction. Sorters define the
order in which hierarchy layers (e.g. roles or frameworks) should be offered
resources by taking “client” objects and some information about
those clients and returning an ordered list of clients.</p>
-<p>Sorters are implemented in C++ and inherit the <code>Sorter</code> class
defined in <code>$MESOS_HOME/src/master/allocator/sorter/sorter.hpp</code>. The
default sorter is <code>DRFSorter</code>, which implements fair sharing and can
be found in
<code>$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp</code>. This
sorter is capable of expressing priorities by specifying weights in
<code>Sorter::add()</code>. Each client’s share is divided by its weight.
For example, a role that has a weight of <code>2</code> will be offered twice
as many resources as a role with weight <code>1</code>.</p>
+<p>Sorters are implemented in C++ and inherit the <code>Sorter</code> class
defined in <code>$MESOS_HOME/src/master/allocator/sorter/sorter.hpp</code>. The
default sorter is <code>DRFSorter</code>, which implements fair sharing and can
be found in
<code>$MESOS_HOME/src/master/allocator/sorter/drf/sorter.hpp</code>. This
sorter is capable of expressing priorities by specifying weights in
<code>Sorter::add()</code>. Each client’s share is divided by its weight.
For example, a role that has a weight of 2 will be offered twice as many
resources as a role with weight 1.</p>
<h2>Wiring up a custom allocator</h2>
Modified:
mesos/site/publish/documentation/latest/app-framework-development-guide/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/app-framework-development-guide/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
---
mesos/site/publish/documentation/latest/app-framework-development-guide/index.html
(original)
+++
mesos/site/publish/documentation/latest/app-framework-development-guide/index.html
Wed Sep 30 07:05:01 2015
@@ -283,17 +283,17 @@ virtual void error(ExecutorDriver* drive
<h2>Install your Framework</h2>
-<p>You need to put your framework somewhere that all slaves on the cluster can
get it from. If you are running HDFS, you can put your executor into HDFS.
Then, you tell Mesos where it is via the <code>ExecutorInfo</code> parameter of
<code>MesosSchedulerDriver</code>’s constructor (e.g. see
src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a
a Protocol Buffer Message class (defined in
<code>include/mesos/mesos.proto</code>), and you set its URI field to something
like “HDFS://path/to/executor/”. Also, you can pass the
<code>frameworks_home</code> configuration option (defaults to:
<code>MESOS_HOME/frameworks</code>) to your <code>mesos-slave</code> daemons
when you launch them to specify where all of your framework executors are
stored (e.g. on an NFS mount that is available to all slaves), then set
<code>ExecutorInfo</code> to be a relative path, and the slave will prepend the
value of frameworks_home to the relative path provided.</p>
+<p>You need to put your framework somewhere that all slaves on the cluster can
get it from. If you are running HDFS, you can put your executor into HDFS.
Then, you tell Mesos where it is via the <code>ExecutorInfo</code> parameter of
<code>MesosSchedulerDriver</code>’s constructor (e.g. see
src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a
Protocol Buffer Message class (defined in
<code>include/mesos/mesos.proto</code>), and you set its URI field to something
like “HDFS://path/to/executor/”. Also, you can pass the
<code>frameworks_home</code> configuration option (defaults to:
<code>MESOS_HOME/frameworks</code>) to your <code>mesos-slave</code> daemons
when you launch them to specify where all of your framework executors are
stored (e.g. on an NFS mount that is available to all slaves), then set
<code>ExecutorInfo</code> to be a relative path, and the slave will prepend the
value of frameworks_home to the relative path provided.</p>
<p>Once you are sure that your executors are available to the mesos-slaves,
you should be able to run your scheduler, which will register with the Mesos
master, and start receiving resource offers!</p>
<h2>Labels</h2>
-<p><code>Labels</code> can be found in the <code>TaskInfo</code>,
<code>DiscoveryInfo</code> and <code>TaskStatus</code>s and
-let’s framework and module writers use Labels to tag and pass
unstructured
-information around Mesos. Labels are free-form key-value pairs supplied by the
-framework scheduler or label decorator hooks. Below is the protobuf definitions
-of labels:</p>
+<p><code>Labels</code> can be found in the <code>FrameworkInfo</code>,
<code>TaskInfo</code>, <code>DiscoveryInfo</code> and
+<code>TaskStatus</code> messages; framework and module writers can use Labels
to tag and
+pass unstructured information around Mesos. Labels are free-form key-value
pairs
+supplied by the framework scheduler or label decorator hooks. Below is the
+protobuf definitions of labels:</p>
<pre><code class="{.proto}"> optional Labels labels = 11;
</code></pre>
Modified:
mesos/site/publish/documentation/latest/attributes-resources/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/attributes-resources/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/attributes-resources/index.html
(original)
+++ mesos/site/publish/documentation/latest/attributes-resources/index.html Wed
Sep 30 07:05:01 2015
@@ -108,7 +108,7 @@ text : [a-zA-Z0-9_/.-]
<h2>Attributes</h2>
-<p>Attributes are key value pairs (where value is optional) that Mesos passes
along when it sends offers to frameworks. An attribute value supports 3
different <em>types</em>: scalar, range or text.</p>
+<p>Attributes are key-value pairs (where value is optional) that Mesos passes
along when it sends offers to frameworks. An attribute value supports 3
different <em>types</em>: scalar, range or text.</p>
<pre><code>attributes : attribute ( ";" attribute )*
Modified: mesos/site/publish/documentation/latest/deploy-scripts/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/deploy-scripts/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/deploy-scripts/index.html (original)
+++ mesos/site/publish/documentation/latest/deploy-scripts/index.html Wed Sep
30 07:05:01 2015
@@ -81,13 +81,21 @@
<p>See our <a href="/community/">community</a> page for more
details.</p>
</div>
<div class="col-md-8">
- <h1>Deploy Scripts</h1>
+ <h1>Deployment Scripts</h1>
-<p>Mesos includes a set of scripts in <code>[install-prefix]/sbin</code> that
can be used to deploy it on a cluster. To use these scripts, you need to create
two configuration files:
<code>[install-prefix]/var/mesos/deploy/masters</code>, which should list the
hostname(s) of the node(s) you want to be your masters (one per line), and
<code>[install-prefix]/var/mesos/deploy/slaves</code>, which should contain a
list of hostnames for your slaves (one per line). You can then start a cluster
with <code>[install-prefix]/sbin/mesos-start-cluster.sh</code> and stop it with
<code>[install-prefix]/sbin/mesos-stop-cluster.sh</code>.</p>
+<p>Mesos includes a set of scripts in <code>[install-prefix]/sbin</code> that
can be used to deploy it on a cluster. To use these scripts, you need to create
two configuration files:</p>
-<p>It is also possible to set environment variables, ulimits, etc that will
affect the master and slave. by editing
<code>[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh</code>. One
particularly useful setting is <code>LIBPROCESS_IP</code>, which tells the
master and slave binaries which IP address to bind to; in some installations,
the default interface that the hostname resolves to is not the machine’s
external IP address, so you can set the right IP through this variable. Besides
the common environment variables of master and slave configured in
<code>[install-prefix/var/mesos/deploy/mesos-deploy-env.sh</code>, it is also
possible to set master specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-master-env.sh</code>, slave
specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-slave-env.sh</code>.</p>
+<ol>
+<li><code>[install-prefix]/var/mesos/deploy/masters</code>, which should list
the hostname(s) of the node(s) you want to be your masters (one per line),
and</li>
+<li><code>[install-prefix]/var/mesos/deploy/slaves</code>, which should
contain a list of hostnames for your slaves (one per line).</li>
+</ol>
-<p>Finally, the deploy scripts do not use ZooKeeper by default. If you want to
use ZooKeeper (/documentation/latest/for multiple masters). Please see the <a
href="high-availability/">High Availability</a> documentation for details.</p>
+
+<p>You can then start a cluster with
<code>[install-prefix]/sbin/mesos-start-cluster.sh</code> and stop it with
<code>[install-prefix]/sbin/mesos-stop-cluster.sh</code>.</p>
+
+<p>It is also possible to set environment variables, ulimits, etc that will
affect the master and slave by editing
<code>[install-prefix]/var/mesos/deploy/mesos-deploy-env.sh</code>. One
particularly useful setting is <code>LIBPROCESS_IP</code>, which tells the
master and slave binaries which IP address to bind to; in some installations,
the default interface that the hostname resolves to is not the machine’s
external IP address, so you can set the right IP through this variable. Besides
the common environment variables of master and slave configured in
<code>[install-prefix/var/mesos/deploy/mesos-deploy-env.sh</code>, it is also
possible to set master specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-master-env.sh</code>, slave
specific environment variables in
<code>[install-prefix]/var/mesos/deploy/mesos-slave-env.sh</code>.</p>
+
+<p>Finally, the deploy scripts do not use ZooKeeper by default. If you want to
configure Mesos to use ZooKeeper to coordinate multiple master nodes, please
see the <a href="/documentation/latest/high-availability/">High
Availability</a> documentation for details.</p>
<h2>Notes</h2>
Modified:
mesos/site/publish/documentation/latest/fetcher-cache-internals/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/fetcher-cache-internals/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/fetcher-cache-internals/index.html
(original)
+++ mesos/site/publish/documentation/latest/fetcher-cache-internals/index.html
Wed Sep 30 07:05:01 2015
@@ -83,7 +83,7 @@
<div class="col-md-8">
<h1>Mesos Fetcher Cache Internals</h1>
-<p>It assumed that readers of this document are well familiar with the
contents of the overview and user guide of the Mesos fetcher in
“fetcher.md”. The present document makes direct references to
notions defined in the former.</p>
+<p>It assumed that readers of this document are familiar with the contents of
the <a href="/documentation/latest/fetcher/">Mesos fetcher user guide</a>. The
present document makes direct references to notions defined in the former.</p>
<h2>Design goals for the initial fetcher cache prototype:</h2>
@@ -97,9 +97,13 @@
</ol>
-<p>For future releases, we foresee additional features:
-1. Automatic refreshing of cache content when a URI’s content has
changed.
-2. Prefetching URIs for subsequent tasks. Prefetching can run in parallel with
task execution.</p>
+<p>For future releases, we foresee additional features:</p>
+
+<ol>
+<li>Automatic refreshing of cache content when a URI’s content has
changed.</li>
+<li>Prefetching URIs for subsequent tasks. Prefetching can run in parallel
with task execution.</li>
+</ol>
+
<h2>How the fetcher cache works</h2>
@@ -138,7 +142,7 @@
<p>The fetcher process uses a private instance of class Cache to represent
what URIs are cached, where the respective cache files are, what stage of
processing they are in, and so on.</p>
-<p>The main data structure to hold all this information is a hashmap from
URI/user combinations to Cache::Entry objects, which each contain information
about an individual cache file on disk. These objects are referenced by
shared_ptr, because they can be addressed by multiple callbacks on behalf of
concurrent fetch attempts while also being held in the hashmap.</p>
+<p>The main data structure to hold all this information is a hashmap from
URI/user combinations to <code>Cache::Entry</code> objects, which each contain
information about an individual cache file on disk. These objects are
referenced by shared_ptr, because they can be addressed by multiple callbacks
on behalf of concurrent fetch attempts while also being held in the hashmap.</p>
<p>A cache entry corresponds directly to a cache file on disk throughout the
entire life time of the latter, including before and after its existence. It
holds all pertinent state to inform about the phase and results of fetching the
corresponding URI.</p>
@@ -150,10 +154,14 @@
<p>The two blue states are essentially the same: no cache file exists. The two
green disk states on the right are also the same.</p>
-<p>The figure only depicts what happens from the point of view of one isolated
fetch run. Any given cache entry can be referenced simultaniously by another
concurrent fetch run. It must not be evicted as long as it is referenced by any
fetching activity. We implement this by reference counting. Every cache entry
has a reference count field that gets incremented at the beginning of its use
by a fetch run and decremented at its end. The latter must happen no matter
whether the run has been successful or whether there has been an error.
Increments happen when:
-- A new cache entry is created. It is immediately referenced.
-- An existing cache entry’s file download is going to be waited for.
-- An existing cache entry has a resident cache file that is going to be
retrieved.</p>
+<p>The figure only depicts what happens from the point of view of one isolated
fetch run. Any given cache entry can be referenced simultaniously by another
concurrent fetch run. It must not be evicted as long as it is referenced by any
fetching activity. We implement this by reference counting. Every cache entry
has a reference count field that gets incremented at the beginning of its use
by a fetch run and decremented at its end. The latter must happen no matter
whether the run has been successful or whether there has been an error.
Increments happen when:</p>
+
+<ul>
+<li>A new cache entry is created. It is immediately referenced.</li>
+<li>An existing cache entry’s file download is going to be waited
for.</li>
+<li>An existing cache entry has a resident cache file that is going to be
retrieved.</li>
+</ul>
+
<p>Every increment is recorded in a list. At the very end of the fetch
procedure, no matter what its outcome is, each entry in the list gets its
reference count decremented.</p>
@@ -165,17 +173,25 @@
<p><img src="images/fetch_flow.jpg?raw=true" alt="Determining Fetcher Actions"
/></p>
-<p>After going through this procedure for each URI, the fetcher process
assembles the gathered list of per-URI actions into a JSON object
(FetcherInfo), which is passed to the mesos-fetcher program in an environment
variable. The possible fetch actions for a URI are shown at the bottom of the
flow chart. After they are determined, the fetcher process invokes
mesos-fetcher.</p>
+<p>After going through this procedure for each URI, the fetcher process
assembles the gathered list of per-URI actions into a JSON object
(<code>FetcherInfo</code>), which is passed to the mesos-fetcher program in an
environment variable. The possible fetch actions for a URI are shown at the
bottom of the flow chart. After they are determined, the fetcher process
invokes mesos-fetcher.</p>
<p>The implementation is oriented at this control flow but its code structure
cannot match it directly, because some of these branches must span multiple
libprocess continuations. There are two layers of futures, one for each of
these phases.</p>
<ol>
-<li>Before making fetcher cache items.</li>
-<li>a) Wait for concurrent downloads for pre-existing cache entries</li>
-<li><p>b) Wait for size fetching combined and then space reservation for new
cache entries.</p></li>
-<li><p>After making fetcher cache items and running mesos-fetcher.</p></li>
+<li> Before making fetcher cache items,
+
+<ol type="a">
+<li>Wait for concurrent downloads for pre-existing cache entries.</li>
+<li>Wait for size fetching combined and then space reservation for new cache
entries.</li>
+</ol>
+</li>
+<li> After making fetcher cache items and running mesos-fetcher,
+
+<ol type="a">
<li>Complete new cache items with success/failure, which as an important
side-effect informs concurrent fetch runs' futures in phase 1/a.</li>
</ol>
+</li>
+</ol>
<p>The futures for phase 1 are not shared outside one fetch run. They
exclusively guard asynchronous operations for the same fetch run. Their type
parameter does not really matter. But each needs to correspond to one URI and
eventual fetch item somehow. Multiple variants have been proposed for this. The
complexity remains about the same.</p>
@@ -192,9 +208,13 @@
<p>The resources named “A” and “B” have been fetched
with caching into sandbox 1 and 2 below. In the course of this, two cache
entries have been created and two files have been downloaded into the cache and
named “1” and “2”. (Cache file names have unique names
that comprise serial numbers.)</p>
-<p>The next figure illustrates the state after fetching a different cached URI
into sandbox 3, which in this case requires evicting a cache-resident file and
its entry. Cache eviction removes cache entries in the order of the least
recently used cache entries. Steps if “A” was fetched before
“B”:
-1. Remove the cache entry for “A” from the fetcher process' cache
entry table. Its faded depiction is supposed to indicate this. This immediately
makes it appear as if the URI has never been cached, even though the cache file
is still around.
-2. Proceed with fetching “C”. This creates a new cache file, which
has a different unique name. (The fetcher process remembers in its cache entry
which file name belongs to which URI.)</p>
+<p>The next figure illustrates the state after fetching a different cached URI
into sandbox 3, which in this case requires evicting a cache-resident file and
its entry. Cache eviction removes cache entries in the order of the least
recently used cache entries. Steps if “A” was fetched before
“B”:</p>
+
+<ol>
+<li>Remove the cache entry for “A” from the fetcher process' cache
entry table. Its faded depiction is supposed to indicate this. This immediately
makes it appear as if the URI has never been cached, even though the cache file
is still around.</li>
+<li>Proceed with fetching “C”. This creates a new cache file,
which has a different unique name. (The fetcher process remembers in its cache
entry which file name belongs to which URI.)</li>
+</ol>
+
<p><img src="images/fetch_evict2.jpg?raw=true" alt="After eviction" /></p>
Modified: mesos/site/publish/documentation/latest/fetcher/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/fetcher/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/fetcher/index.html (original)
+++ mesos/site/publish/documentation/latest/fetcher/index.html Wed Sep 30
07:05:01 2015
@@ -83,8 +83,7 @@
<div class="col-md-8">
<h1>Mesos Fetcher</h1>
-<p>Experimental support for the Mesos fetcher <em>cache</em> is introduced in
-Mesos 0.23.0.</p>
+<p>Mesos 0.23.0 introduced experimental support for the Mesos <em>fetcher
cache</em>.</p>
<p>In this context we loosely regard the term “downloading” as to
include copying
from local file systems.</p>
@@ -94,7 +93,7 @@ from local file systems.</p>
<p>The Mesos fetcher is a mechanism to download resources into the sandbox
directory of a task in preparation of running the task. As part of a TaskInfo
message, the framework ordering the task’s execution provides a list of
-CommandInfo::URI protobuf values, which becomes the input to the Mesos
fetcher.</p>
+<code>CommandInfo::URI</code> protobuf values, which becomes the input to the
Mesos fetcher.</p>
<p>By default, each requested URI is downloaded directly into the sandbox
directory
and repeated requests for the same URI leads to downloading another copy of the
@@ -109,7 +108,7 @@ coordinates all fetch actions. Every sla
fetcher instance that is used by every kind of containerizer (except the
external containerizer variant, which is responsible for its own approach to
fetching).</p></li>
-<li><p>The external program “mesos-fetcher” that is invoked by the
former. It
+<li><p>The external program <code>mesos-fetcher</code> that is invoked by the
former. It
performs all network and disk operations except file deletions and file size
queries for cache-internal bookkeeping. It is run as an external OS process in
order to shield the slave process from I/O-related hazards. It takes
@@ -120,14 +119,14 @@ detailed fetch action descriptions.</p><
<h2>The fetch procedure</h2>
-<p>Frameworks launch tasks by calling the scheduler driver method
launchTasks(),
-passing CommandInfo protobuf structures as arguments. This type of structure
+<p>Frameworks launch tasks by calling the scheduler driver method
<code>launchTasks()</code>,
+passing <code>CommandInfo</code> protobuf structures as arguments. This type
of structure
specifies (among other things) a command and a list of URIs that need to be
“fetched” into the sandbox directory on the the slave node as a
precondition for
task execution. Hence, when the slave receives a request go launch a task, it
calls upon its fetcher, first, to provision the specified resources into the
sandbox directory. If fetching fails, the task is not started and the reported
-task status is TASK_FAILED.</p>
+task status is <code>TASK_FAILED</code>.</p>
<p>All URIs requested for a given task are fetched sequentially in a single
invocation of mesos-fetcher. Here, avoiding download concurrency reduces the
@@ -138,7 +137,7 @@ active concurrently due to multiple task
<p>Before mesos-fetcher is started, the specific fetch actions to be performed
for
each URI are determined based on the following protobuf structure. (See
-“include/mesos/mesos.proto” for more details.)</p>
+<code>include/mesos/mesos.proto</code> for more details.)</p>
<pre><code>message CommandInfo {
message URI {
@@ -169,7 +168,7 @@ the affected task.</p>
<p>If a user name is specified either way, the fetcher first validates that it
is
in fact a valid user name on the slave. If it is not, fetching fails right
here.
Otherwise, the sandbox directory is assigned to the specified user as owner
-(using chown) at the end of the fetch procedure, before task execution
begins.</p>
+(using <code>chown</code>) at the end of the fetch procedure, before task
execution begins.</p>
<p>The user name in play has an important effect on caching. Caching is
managed on
a per-user base, i.e. the combination of user name and “uri”
uniquely
@@ -259,7 +258,7 @@ the URI.</p>
<ul>
<li>Local file sizes are probed with systems calls (that follow symbolic
links).</li>
<li>HTTP/HTTPS URIs are queried for the “content-length” field in
the header. This
-is performed by CURL. The reported asset size must be greater than zero or
+is performed by <code>curl</code>. The reported asset size must be greater
than zero or
the URI is deemed invalid.</li>
<li>FTP/FTPS is not supported at the time of writing.</li>
<li>Everything else is queried by the local HDFS client.</li>
@@ -355,6 +354,10 @@ running the present task, right after fe
</ul>
+<h2>Implementation Details</h2>
+
+<p>The <a href="/documentation/latest/fetcher-cache-internals/">Mesos Fetcher
Cache Internals</a> describes how the fetcher cache is implemented.</p>
+
</div>
</div>
Modified: mesos/site/publish/documentation/latest/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/index.html (original)
+++ mesos/site/publish/documentation/latest/index.html Wed Sep 30 07:05:01 2015
@@ -94,10 +94,12 @@
<h2>Running Mesos</h2>
<ul>
+<li><a href="/documentation/latest/getting-started/">Getting Started</a> for
basic instructions on compiling and installing Mesos.</li>
<li><a href="/documentation/latest/configuration/">Configuration</a> for
command-line arguments.</li>
<li><a href="/documentation/latest/mesos-containerizer/">Mesos
Containerizer</a> default containerizer, supports both Linux and POSIX
systems.</li>
<li><a href="/documentation/latest/docker-containerizer/">Docker
Containerizer</a> for launching a Docker image as a Task, or as an
Executor.</li>
<li><a href="/documentation/latest/external-containerizer/">External
Containerizer</a> for custom containerization implementations.</li>
+<li><a href="/documentation/latest/authentication/">Framework
Authentication</a></li>
<li><a href="/documentation/latest/authorization/">Framework
Authorization</a></li>
<li><a href="/documentation/latest/framework-rate-limiting/">Framework Rate
Limiting</a></li>
<li><a href="/documentation/latest/logging-and-debugging/">Logging and
Debugging</a> for viewing Mesos and framework logs.</li>
@@ -112,10 +114,21 @@
</ul>
+<h2>Advanced Features</h2>
+
+<ul>
+<li><a href="/documentation/attributes-resources/">Attributes and
Resources</a> for how to describe the slaves that comprise a cluster.</li>
+<li><a href="/documentation/latest/fetcher/">Fetcher Cache</a> for how to
configure the Mesos fetcher cache.</li>
+<li><a href="/documentation/latest/oversubscription/">Oversubscription</a> for
how to configure Mesos to take advantage of unused resources to launch
“best-effort” tasks.</li>
+<li><a href="/documentation/latest/persistent-volume/">Persistent Volume</a>
for how to allow tasks to access persistent storage resources.</li>
+<li><a href="/documentation/latest/reservation/">Reservation</a> for how to
configure Mesos to allow slaves to reserve resources.</li>
+</ul>
+
+
<h2>Running Mesos Frameworks</h2>
<ul>
-<li><a href="/documentation/latest/mesos-frameworks/">Mesos frameworks</a> for
a list of apps built on top of Mesos, and instructions on how to run them.</li>
+<li><a href="/documentation/latest/mesos-frameworks/">Mesos frameworks</a> for
a list of apps built on top of Mesos and instructions on how to run them.</li>
</ul>
@@ -124,16 +137,18 @@
<ul>
<li><a href="/documentation/latest/app-framework-development-guide/">Framework
Development Guide</a> describes how to build applications on top of Mesos.</li>
<li><a href="/documentation/latest/reconciliation/">Reconciliation</a> for
ensuring a framework’s state remains eventually consistent in the face of
failures.</li>
+<li><a href="/documentation/latest/scheduler_http_api/">Scheduler HTTP API</a>
describes the new HTTP API for communication between schedulers and the Mesos
master.</li>
<li><a href="/api/latest/java/">Javadoc</a> documents the Mesos Java API.</li>
<li><a href="/api/latest/c++/namespacemesos.html">Doxygen</a> documents the
Mesos C++ API.</li>
<li><a href="/documentation/latest/tools/">Developer Tools</a> for hacking on
Mesos or writing frameworks.</li>
</ul>
-<h2>Mesos Modules</h2>
+<h2>Extending Mesos</h2>
<ul>
<li><a href="/documentation/latest/modules/">Mesos Modules</a> for specifying
Mesos modules for master, slave and tests.</li>
+<li><a href="/documentation/latest/allocation-module/">Allocation Modules</a>
for how to write custom resource allocators.</li>
</ul>
@@ -141,7 +156,7 @@
<ul>
<li><a href="/documentation/latest/reporting-a-bug/">Reporting an Issue,
Improvement, or Feature</a> for getting started with JIRA.</li>
-<li><a href="/documentation/latest/submitting-a-patch/">Submitting a Patch</a>
for getting started with ReviewBoard, and our tooling around it.</li>
+<li><a href="/documentation/latest/submitting-a-patch/">Submitting a Patch</a>
for getting started with ReviewBoard and our tooling around it.</li>
<li><a href="/documentation/latest/mesos-testing-patterns/">Testing
Patterns</a> for tips and tricks used in Mesos tests.</li>
<li><a href="/documentation/latest/effective-code-reviewing/">Effective Code
Reviewing</a> guidelines, tips, and learnings for how to do effective code
reviews.</li>
<li><a
href="/documentation/latest/engineering-principles-and-practices/">Engineering
Principles and Practices</a> to serve as a shared set of project-level values
for the community.</li>
Modified:
mesos/site/publish/documentation/latest/mesos-c++-style-guide/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/mesos-c%2B%2B-style-guide/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/mesos-c++-style-guide/index.html
(original)
+++ mesos/site/publish/documentation/latest/mesos-c++-style-guide/index.html
Wed Sep 30 07:05:01 2015
@@ -83,7 +83,17 @@
<div class="col-md-8">
<h1>Mesos C++ Style Guide</h1>
-<p>The Mesos codebase follows the <a
href="http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml";>Google
C++ Style Guide</a> with the following differences:</p>
+<p>The Mesos codebase follows the <a
href="http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml";>Google
C++ Style Guide</a> with some notable differences, as described below. Note
that the <a href="/documentation/latest/clang-format/">clang-format</a> tool
can be helpful to ensure that some of the mechanical style rules are obeyed.</p>
+
+<h2>Scoping</h2>
+
+<h3>Namespaces</h3>
+
+<ul>
+<li>We avoid <code>using namespace foo</code> statements as it is not explicit
about which symbols are pulled in, and it can often pull in a lot of symbols,
which sometimes lead to conflicts.</li>
+<li>It is OK to use namespace aliases to help pull in sub-namespaces, such as
<code>namespace http = process::http;</code>. These should only be present at
the top of the .cpp file.</li>
+</ul>
+
<h2>Naming</h2>
@@ -129,13 +139,6 @@ void Slave::statusUpdate(StatusUpdate up
</ul>
-<h3>Namespace Names</h3>
-
-<ul>
-<li>We do not use namespace aliases.</li>
-</ul>
-
-
<h2>Strings</h2>
<ul>
Modified: mesos/site/publish/documentation/latest/oversubscription/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/oversubscription/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/oversubscription/index.html
(original)
+++ mesos/site/publish/documentation/latest/oversubscription/index.html Wed Sep
30 07:05:01 2015
@@ -131,7 +131,7 @@ resources such as cpu shares, bandwidth,
<ul>
<li>(5) Frameworks can choose to launch tasks on revocable resources by using
-the regular launchTasks() API. To safe-guard frameworks that are not
+the regular <code>launchTasks()</code> API. To safe-guard frameworks that are
not
designed to deal with preemption, only frameworks registering with the
<code>REVOCABLE_RESOURCES</code> capability set in its framework info will
receive offers
with revocable resources. Further more, revocable resources cannot be
@@ -143,7 +143,7 @@ disk resources.</li>
<h3>Task launch</h3>
<ul>
-<li>The revocable task is launched as usual when the runTask request is
received
+<li>The revocable task is launched as usual when the <code>runTask</code>
request is received
on the slave. The resources will still be marked as revocable and isolators
can take appropriate actions, if certain resources need to be setup differently
for revocable and regular tasks.</li>
@@ -300,7 +300,7 @@ the type of action to perform.</p>
}
</code></pre>
-<h2>Configuring Mesos for oversubscription</h2>
+<h2>Configuring oversubscription</h2>
<p>Five new flags has been added to the slave:</p>
Modified: mesos/site/publish/documentation/latest/persistent-volume/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/persistent-volume/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/persistent-volume/index.html
(original)
+++ mesos/site/publish/documentation/latest/persistent-volume/index.html Wed
Sep 30 07:05:01 2015
@@ -81,7 +81,7 @@
<p>See our <a href="/community/">community</a> page for more
details.</p>
</div>
<div class="col-md-8">
- <h1>Persistent Volume</h1>
+ <h1>Persistent Volumes</h1>
<p>Mesos provides a mechanism to create a persistent volume from disk
resources. When launching a task, you can create a volume that exists outside
Modified: mesos/site/publish/documentation/latest/reconciliation/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/reconciliation/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/reconciliation/index.html (original)
+++ mesos/site/publish/documentation/latest/reconciliation/index.html Wed Sep
30 07:05:01 2015
@@ -85,17 +85,17 @@
<p>There’s no getting around it, <strong>frameworks on Mesos are
distributed systems</strong>.</p>
-<p><strong>Distributed systems must deal with failures</strong>, and
partitions (the two are
+<p><strong>Distributed systems must deal with failures</strong> and partitions
(the two are
indistinguishable from a system’s perspective).</p>
<p>Concretely, what does this mean for frameworks? Mesos uses an actor-like
-<strong>message passing programming model, in which messages are delivered
-at-most-once</strong>. (Exceptions to this include task status updates, most of
+<strong>message passing</strong> programming model, in which messages are
delivered
+<strong>at-most-once</strong>. (Exceptions to this include task status
updates, most of
which are delivered at-least-once through the use of acknowledgements).
<strong>The messages passed between the master and the framework are therefore
susceptible to be dropped, in the presence of failures</strong>.</p>
-<p>When these non-reliable messages are dropped, inconsistent state can arise
+<p>When these unreliable messages are dropped, inconsistent state can arise
between the framework and Mesos.</p>
<p>As a simple example, consider a launch task request sent by a framework.
Modified: mesos/site/publish/documentation/latest/reporting-a-bug/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/reporting-a-bug/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/reporting-a-bug/index.html
(original)
+++ mesos/site/publish/documentation/latest/reporting-a-bug/index.html Wed Sep
30 07:05:01 2015
@@ -88,7 +88,7 @@
<ul>
<li>We track all issues via Apache’s hosted <a
href="https://issues.apache.org/jira/browse/MESOS";>JIRA issue tracker</a>, so
if you don’t already have an account you’ll need to register for
one. It’s quick and easy.</li>
<li>A JIRA should be created for every task, feature, bug-fix, etc. This makes
it easy to track progress.</li>
-<li>If you are planning to work on the ticket, please assign the JIRA issue to
yourself before you start working on it. This helps to avoid duplication of
work, and alerts anyone who is following that issue. It is highly encouraged to
start a discussion with others in the community about how to solve the
issue.</li>
+<li>If you are planning to work on the ticket, please assign the JIRA issue to
yourself before you start working on it. This helps to avoid duplication of
work and alerts anyone who is following that issue. It is highly encouraged to
start a discussion with others in the community about how to solve the issue.
The <a href="/documentation/latest/submitting-a-patch/">contribution
guidelines</a> discuss how to contribute to Mesos development in more
detail.</li>
</ul>
Modified: mesos/site/publish/documentation/latest/scheduler_http_api/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/scheduler_http_api/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/scheduler_http_api/index.html
(original)
+++ mesos/site/publish/documentation/latest/scheduler_http_api/index.html Wed
Sep 30 07:05:01 2015
@@ -537,7 +537,7 @@ HTTP/1.1 202 Accepted
<h2>Master detection</h2>
-<p>Mesos has a high-availability mode that uses multiple Mesos masters; one
active master (/documentation/latest/called the leader or leading master) and
several standbys in case it fails. The masters elect the leader, with ZooKeeper
coordinating the election. For more details please refer to the <a
href="high-availability/">documentation</a>.</p>
+<p>Mesos has a high-availability mode that uses multiple Mesos masters; one
active master (/documentation/latest/called the leader or leading master) and
several standbys in case it fails. The masters elect the leader, with ZooKeeper
coordinating the election. For more details please refer to the <a
href="/documentation/latest/high-availability/">documentation</a>.</p>
<p>Schedulers are expected to make HTTP requests to the leading master. If
requests are made to a non-leading master a âHTTP 307 Temporary Redirectâ
will be received with the âLocationâ header pointing to the leading
master.</p>
Modified: mesos/site/publish/documentation/latest/submitting-a-patch/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/submitting-a-patch/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/submitting-a-patch/index.html
(original)
+++ mesos/site/publish/documentation/latest/submitting-a-patch/index.html Wed
Sep 30 07:05:01 2015
@@ -90,26 +90,46 @@
</ul>
-<h3>Before you submit</h3>
+<h3>Before you start writing code</h3>
<ol>
<li><p>Check out the code from the Apache repository via Git. Instructions are
on the <a href="http://mesos.apache.org/gettingstarted/";>Getting Started</a>
page.</p></li>
-<li><p>Join the dev, issues, reviews and builds mailing lists by sending
emails to [email protected], [email protected],
[email protected] and [email protected]
respectively.</p></li>
-<li><p>Find a JIRA that is currently unassigned that you want to work on at <a
href="https://issues.apache.org/jira/browse/MESOS";>JIRA issue tracker</a>, or
create your own (you’ll need a JIRA account for this, see below)!</p>
+<li><p>Join the dev, issues, reviews and builds mailing lists by sending
emails to <a
href="mailto:[email protected]";>[email protected]</a>,
<a
href="mailto:[email protected]";>[email protected]</a>,
<a
href="mailto:[email protected]";>[email protected]</a>
and <a
href="mailto:[email protected]";>[email protected]</a>
respectively.</p></li>
+<li><p>Find a JIRA issue that is currently unassigned that you want to work on
at <a href="https://issues.apache.org/jira/browse/MESOS";>JIRA issue
tracker</a>, or create your own (you’ll need a JIRA account for this, see
below)!</p>
<ol>
<li>This could be a JIRA representing a bug (possibly a bug that you
encountered and reported, e.g. when trying to build) or a new feature.</li>
+<li>Prefer working on issues marked as “<a
href="https://issues.apache.org/jira/browse/MESOS-1?jql=project%20%3D%20MESOS%20AND%20status%20%3D%20Accepted";>Accepted</a>”,
rather than merely “Open”. If an issue has been accepted, it means
at least one Mesos developer thought that the ideas proposed in the issue are
worth pursuing further.</li>
+<li>Issues marked with the “<a
href="https://issues.apache.org/jira/browse/MESOS-1?jql=project%20%3D%20MESOS%20AND%20status%20%3D%20Accepted%20AND%20labels%20%3D%20newbie";>newbie</a>”
label can be good candidates for “starter” projects.</li>
</ol>
</li>
<li><p>Assign the JIRA to yourself. To do this, you will need:</p>
<ol>
<li>An Apache JIRA user account (sign up for one <a
href="https://issues.apache.org/jira/secure/Signup!default.jspa";>here</a>).</li>
-<li>You need to be added to the list of Mesos “contributors” by a
Mesos committer (send email to [email protected]) in order to be assigned
(or to assign yourself) to a JIRA issue.</li>
+<li>You need to be added to the list of Mesos “contributors” by a
Mesos committer (send email to <a
href="mailto:[email protected]";>[email protected]</a>) in order to be
assigned (or to assign yourself) to a JIRA issue.</li>
+</ol>
+</li>
+<li><p>Formulate a plan for resolving the issue. Propose your plan via
comments in the JIRA.</p></li>
+<li><p>Find a <strong>shepherd</strong> to collaborate on your patch. A
shepherd is a Mesos committer that will work with you to give you feedback on
your proposed design, and to eventually commit your change into the Mesos
source tree.</p>
+
+<ol>
+<li>To find a shepherd, email the dev mailing list (include a link to your
JIRA issue). You can also try asking by adding a comment to your JIRA
issue.</li>
+<li>You can also find a shepherd by asking the developers on IRC
(/documentation/latest/in the <a href="irc://irc.freenode.net/mesos">mesos
channel</a> on <a href="https://freenode.net";>Freenode</a>). You can find the
current list of committers <a href="committers/">here</a>: a developer that has
previously worked on the component you are modifying might be a good candidate
shepherd.</li>
+</ol>
+</li>
+</ol>
+
+
+<h3>Create your patch</h3>
+
+<ol>
+<li><p>Create one or more test cases to exercise the bug or the feature (the
Mesos team uses <a
href="http://en.wikipedia.org/wiki/Test-driven_development";>test-driven
development</a>). Before you start coding, make sure these test cases all
fail.</p>
+
+<ol>
+<li>The <a href="/documentation/latest/mesos-testing-patterns/">testing
patterns</a> page has some suggestions for writing test cases.</li>
</ol>
</li>
-<li><p>Formulate a plan for resolving the issue, propose your plan via
comments in the JIRA.</p></li>
-<li><p>Create one or more test cases to exercise the bug or the feature (the
Mesos team uses <a
href="http://en.wikipedia.org/wiki/Test-driven_development";>test-driven
development</a>), before you start coding, make sure these test cases all
fail.</p></li>
<li><p>Make your changes to the code (using whatever IDE/editor you choose) to
actually fix the bug or implement the feature.</p>
<ol>
@@ -127,10 +147,10 @@
</li>
</ol>
</li>
-<li><p>Make sure all of your test cases now pass.</p>
+<li><p>Make sure that all of the unit tests pass, including the new test cases
you have added: <code>make check</code>.</p>
<ol>
-<li><code>make check</code></li>
+<li>To execute a single unit test (helpful when trying to debug a test case
failure), use something like: <code>make check
GTEST_FILTER="HTTPTest.Delete"</code>.</li>
</ol>
</li>
<li><p>Make sure to pull in any changes that have been committed to master
branch. Using Git, do this via something like:</p>
@@ -151,20 +171,23 @@
<li><p>You’re ready to submit your patch for review!</p>
<ol>
-<li>Log in or create an account at <a href="http://reviews.apache.org";>Apache
Review Board</a></li>
+<li>Log in or create an account at <a href="http://reviews.apache.org";>Apache
Review Board</a>.</li>
<li>The easiest (and recommended) way to submit reviews is through
<code>post-reviews.py</code> a wrapper around post-review.</li>
-<li>First, install RBTools. <a
href="https://www.reviewboard.org/docs/rbtools/dev/";>See Instructions</a></li>
-<li>Configure post-review. The easiest method is to symlink to the sample
config: ln -s support/reviewboardrc .reviewboardrc</li>
-<li>Log into Review Board from the command line: run <code>rbt
status</code></li>
+<li>First, install RBTools. <a
href="https://www.reviewboard.org/docs/rbtools/dev/";>See Instructions</a>.</li>
+<li>Configure post-review. The easiest method is to symlink to the sample
config: <code>ln -s support/reviewboardrc .reviewboardrc</code>.</li>
+<li>Log into Review Board from the command line: run <code>rbt
status</code>.</li>
<li>From your local branch run <code>support/post-reviews.py</code>.</li>
<li>Note that <code>post-reviews.py</code> creates a new review for every
commit on your branch that is different from the <code>master</code>.</li>
-<li>Be sure to add your JIRA issue id (e.g. MESOS-01) to the field labeled
“Bugs” (this will automatically link)</li>
-<li>Under “Description” in addition to details about your changes,
include a description of any documentation pages that need to be added, or are
affected by your changes (e.g. did you change or add any configuration
options/flags? Did you add a new binary?)</li>
+<li>Be sure to add your JIRA issue id (e.g. MESOS-1) to the field labeled
“Bugs” (this will automatically link).</li>
+<li>Add your shepherd under the “People” field, in the
“Reviewers” section. You should also include other Mesos community
members who have contributed to the discussion of your proposed change.</li>
+<li> Under “Description” in addition to details about your
changes, include a description of any documentation pages that need to be
added, or are affected by your changes (e.g. did you change or add any
configuration options/flags? Did you add a new binary?)</li>
+<li> Under “Testing Done”, explain what new tests you have
created, what tests were modified, and what procedures you went through to test
your changes.</li>
</ol>
</li>
<li><p>Wait for a code review from another Mesos developer via Review Board,
address their feedback and upload updated patches until you receive a
“Ship It” from a Mesos committer.</p>
<ol>
+<li>If you don’t receive any feedback, contact your shepherd to remind
them. While the committers try their best to provide prompt feedback on
proposed changes, they are busy and sometimes a patch gets overlooked.</li>
<li>When addressing feedback, adjust your existing commit(s) instead of
creating new commits, otherwise <code>post-reviews.py</code> will create a new
review (<code>git rebase -i</code> is your friend).</li>
<li>Review Board comments should be used for code-specific discussions, and
JIRA comments for bigger-picture design discussions.</li>
<li>Always respond to each RB comment that you address directly (i.e. each
comment can be responded to directly) with either “Done.” or a
comment explaining how you addressed it.</li>
@@ -179,7 +202,7 @@
<h2>Style Guides</h2>
<ul>
-<li>For patches to the core, we ask that you follow the <a
href="/documentation/latest/mesos-c++-style-guide/">Mesos C++ Style
Guide</a></li>
+<li>For patches to the core, we ask that you follow the <a
href="/documentation/latest/mesos-c++-style-guide/">Mesos C++ Style
Guide</a>.</li>
</ul>
Modified: mesos/site/publish/documentation/latest/tools/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/tools/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/tools/index.html (original)
+++ mesos/site/publish/documentation/latest/tools/index.html Wed Sep 30
07:05:01 2015
@@ -103,6 +103,7 @@
<p>If you want to hack on Mesos or write a new framework, these tools will
help.</p>
<ul>
+<li><a href="/documentation/latest/clang-format/">clang-format</a> to
automatically apply some of the style rules dictated by the <a
href="/documentation/latest/mesos-c++-style-guide/">Mesos C++ Style
Guide</a>.</li>
<li><a href="https://github.com/mesosphere/mesos-go";>Go Bindings and
Examples</a> Write a Mesos framework in Go! Comes with an example scheduler and
executor.</li>
<li><a href="https://github.com/mesosphere/scala-sbt-mesos-framework.g8";>Mesos
Framework giter8 Template</a> This is a giter8 template. The result of applying
this template is a bare-bones Apache Mesos framework in Scala using SBT for
builds and Vagrant for testing on a singleton cluster.</li>
<li><a href="https://gist.github.com/guenter/7471695";>Scala Hello World</a> A
simple Mesos “Hello World”: downloads and starts a web server on
every node in the cluster.</li>
Modified: mesos/site/publish/documentation/mesos-c++-style-guide/index.html
URL:
http://svn.apache.org/viewvc/mesos/site/publish/documentation/mesos-c%2B%2B-style-guide/index.html?rev=1705971&r1=1705970&r2=1705971&view=diff
==============================================================================
--- mesos/site/publish/documentation/mesos-c++-style-guide/index.html (original)
+++ mesos/site/publish/documentation/mesos-c++-style-guide/index.html Wed Sep
30 07:05:01 2015
@@ -83,7 +83,17 @@
<div class="col-md-8">
<h1>Mesos C++ Style Guide</h1>
-<p>The Mesos codebase follows the <a
href="http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml";>Google
C++ Style Guide</a> with the following differences:</p>
+<p>The Mesos codebase follows the <a
href="http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml";>Google
C++ Style Guide</a> with some notable differences, as described below. Note
that the <a href="/documentation/latest/clang-format/">clang-format</a> tool
can be helpful to ensure that some of the mechanical style rules are obeyed.</p>
+
+<h2>Scoping</h2>
+
+<h3>Namespaces</h3>
+
+<ul>
+<li>We avoid <code>using namespace foo</code> statements as it is not explicit
about which symbols are pulled in, and it can often pull in a lot of symbols,
which sometimes lead to conflicts.</li>
+<li>It is OK to use namespace aliases to help pull in sub-namespaces, such as
<code>namespace http = process::http;</code>. These should only be present at
the top of the .cpp file.</li>
+</ul>
+
<h2>Naming</h2>
@@ -129,13 +139,6 @@ void Slave::statusUpdate(StatusUpdate up
</ul>
-<h3>Namespace Names</h3>
-
-<ul>
-<li>We do not use namespace aliases.</li>
-</ul>
-
-
<h2>Strings</h2>
<ul>
