Repository: mesos-site Updated Branches: refs/heads/asf-site 72fbb09dd -> d81c19afa
Updated the website built from mesos SHA: 34d92d8. Project: http://git-wip-us.apache.org/repos/asf/mesos-site/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos-site/commit/d81c19af Tree: http://git-wip-us.apache.org/repos/asf/mesos-site/tree/d81c19af Diff: http://git-wip-us.apache.org/repos/asf/mesos-site/diff/d81c19af Branch: refs/heads/asf-site Commit: d81c19afa96a137064fb224c8b8ac1a6ef1c502a Parents: 72fbb09 Author: jenkins <bui...@apache.org> Authored: Fri Apr 13 22:27:47 2018 +0000 Committer: jenkins <bui...@apache.org> Committed: Fri Apr 13 22:27:47 2018 +0000 ---------------------------------------------------------------------- content/blog/feed.xml | 2 +- .../index.html | 2 +- .../latest/replicated-log-internals/index.html | 12 ++++++++++-- content/documentation/latest/upgrades/index.html | 7 +++++++ .../documentation/replicated-log-internals/index.html | 12 ++++++++++-- content/documentation/upgrades/index.html | 7 +++++++ 6 files changed, 36 insertions(+), 6 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/mesos-site/blob/d81c19af/content/blog/feed.xml ---------------------------------------------------------------------- diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 966a07f..254d5e4 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -292,7 +292,7 @@ To learn more about CSI work in Mesos, you can dig into the design document < </ul> -<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="&#109;&#97;&#105;&#108;&#116;&#111;&#58;&#100;&#101;&#118;&#64;&#x61;&#x70;&#x61;&#99;&#x68;&#x65;&#x2e;&#x6d;&#101;&#x73;&#111;&#x73;&#x2e;&#x6f;&#x72;&#103;">&#100;&#x65;&#x76;&#x40;&#x61;&#x70;&#x61;&#x63;&#x68;&#x65;&#x2e;&#x6d;&#101;&#115;&#x6f;&#x73;&#x2e;&#111;&#114;&#103;</a>.</p> +<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="&#109;&#x61;&#x69;&#x6c;&#x74;&#x6f;&#58;&#x64;&#x65;&#x76;&#x40;&#x61;&#x70;&#97;&#99;&#x68;&#x65;&#x2e;&#109;&#x65;&#x73;&#x6f;&#x73;&#46;&#x6f;&#x72;&#x67;">&#x64;&#x65;&#118;&#x40;&#x61;&#112;&#97;&#99;&#x68;&#x65;&#46;&#109;&#101;&#x73;&#x6f;&#x73;&#46;&#111;&#x72;&#x67;</a>.</p> </content> </entry> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/d81c19af/content/blog/performance-working-group-progress-report/index.html ---------------------------------------------------------------------- diff --git a/content/blog/performance-working-group-progress-report/index.html b/content/blog/performance-working-group-progress-report/index.html index f54e270..e6f7872 100644 --- a/content/blog/performance-working-group-progress-report/index.html +++ b/content/blog/performance-working-group-progress-report/index.html @@ -238,7 +238,7 @@ </ul> -<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="mailto:dev@apache.mesos.org">dev@apache.mesos.org</a>.</p> +<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="mailto:dev@apache.mesos.org">dev@apache.mesos.org</a>.</p> </div> </div> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/d81c19af/content/documentation/latest/replicated-log-internals/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/latest/replicated-log-internals/index.html b/content/documentation/latest/replicated-log-internals/index.html index fd77b6a..e26d278 100644 --- a/content/documentation/latest/replicated-log-internals/index.html +++ b/content/documentation/latest/replicated-log-internals/index.html @@ -143,7 +143,7 @@ <h3>Reaching consensus for a single log entry</h3> -<p>A Paxos round can help all replicas reach consensus on a single log entry’s value. It has two phases: a promise phase and a write phase. Note that we are using slightly different terminology from the <a href="https://research.microsoft.com/en-us/um/people/lamport/pubs/paxes-simple.pdf";>original Paxos paper</a>. In our implementation, the <em>prepare</em> and <em>accept</em> phases in the original paper are referred to as the <em>promise</em> and <em>write</em> phases, respectively. Consequently, a prepare request (response) is referred to as a promise request (response), and an accept request (response) is referred to as a write request (response).</p> +<p>A Paxos round can help all replicas reach consensus on a single log entry’s value. It has two phases: a promise phase and a write phase. Note that we are using slightly different terminology from the <a href="https://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf";>original Paxos paper</a>. In our implementation, the <em>prepare</em> and <em>accept</em> phases in the original paper are referred to as the <em>promise</em> and <em>write</em> phases, respectively. Consequently, a prepare request (response) is referred to as a promise request (response), and an accept request (response) is referred to as a write request (response).</p> <p>To append value <em>X</em> to the log at position <em>p</em>, the coordinator first broadcasts a promise request to all replicas with proposal number <em>n</em>, asking replicas to promise that they will not respond to any request (promise/write request) with a proposal number lower than <em>n</em>. We assume that <em>n</em> is higher than any other previously used proposal number, and will explain how we do this later.</p> @@ -213,12 +213,20 @@ <h3>Auto initialization</h3> -<p>Since we don’t allow an empty replica (a replica in EMPTY status) to respond to requests from coordinators, that raises a question for bootstrapping because initially, each replica is empty. The replicated log provides two choices here. One choice is to use a tool (`mesos-log) to explicitly initialize the log on each replica by setting the replica’s status to VOTING, but that requires an extra step when setting up an application.</p> +<p>Since we don’t allow an empty replica (a replica in EMPTY status) to respond to requests from coordinators, that raises a question for bootstrapping because initially, each replica is empty. The replicated log provides two choices here. One choice is to use a tool (<code>mesos-log</code>) to explicitly initialize the log on each replica by setting the replica’s status to VOTING, but that requires an extra step when setting up an application.</p> <p>The other choice is to do automatic initialization. Our idea is: we allow a replica in EMPTY status to become VOTING immediately if it finds all replicas are in EMPTY status. This is based on the assumption that the only time <em>all</em> replicas are in EMPTY status is during start-up. This may not be true if a catastrophic failure causes all replicas to lose their durable state, and that’s exactly the reason we allow conservative users to disable auto-initialization.</p> <p>To do auto-initialization, if we use a single-phase protocol and allow a replica to directly transit from EMPTY status to VOTING status, we may run into a state where we cannot make progress even if all replicas are in EMPTY status initially. For example, say the quorum size is 2. All replicas are in EMPTY status initially. One replica will first set its status to VOTING because if finds all replicas are in EMPTY status. After that, neither the VOTING replica nor the EMPTY replicas can make progress. To solve this problem, we use a two-phase protocol and introduce an intermediate transient status (STARTING) between EMPTY and VOTING status. A replica in EMPTY status can transit to STARTING status if it finds all replicas are in either EMPTY or STARTING status. A replica in STARTING status can transit to VOTING status if it finds all replicas are in either STARTING or VOTING status. In that way, in our previous example, all replicas will be in STARTING status before any of them can transit to VOTING status.</p> +<h2>Non-leading VOTING replica catch-up</h2> + +<p>Starting with Mesos 1.5.0 it is possible to perform eventually consistent reads from a non-leading VOTING log replica. This makes possible to do additional work on non-leading framework replicas, e.g. offload some reading from a leader to standbys reduce failover time by keeping in-memory storage represented by the replicated log “hot”.</p> + +<p>To serve eventually consistent reads a replica needs to perform <em>catch-up</em> to recover the latest log state in a manner similar to how it is done during <a href="#catch-up">EMPTY replica recovery</a>. After that the recovered positions can be replayed without fear of seeing “holes”.</p> + +<p>A truncation can take place during the non-leading replica catch-up. The replica may try to fill the truncated position if truncation happens after the replica has recovered <em>begin</em> and <em>end</em> positions, which may lead to producing inconsistent data during log replay. In order to protect against it we use a special tombstone flag that signals to the replica that the position was truncated and <em>begin</em> needs to be adjusted. The replica is not blocked from truncations during or after catching-up, which means that the user may need to retry the catch-up procedure if positions that were recovered became truncated during log replay.</p> + <h2>Future work</h2> <p>Currently, replicated log does not support dynamic quorum size change, also known as <em>reconfiguration</em>. Supporting reconfiguration would allow us more easily to add, move or swap hosts for replicas. We plan to support reconfiguration in the future.</p> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/d81c19af/content/documentation/latest/upgrades/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/latest/upgrades/index.html b/content/documentation/latest/upgrades/index.html index 5b6f0c5..7c465ae 100644 --- a/content/documentation/latest/upgrades/index.html +++ b/content/documentation/latest/upgrades/index.html @@ -546,6 +546,13 @@ changed. Note that if this feature is used, the master version is required to be </ul> +<p><a name="1-5-x-log-reader-catchup"></a></p> + +<ul> +<li>A new <code>catchup()</code> method has been added to the replicated log reader API. The method allows to catch-up positions missing in the local non-leading replica to allow safe eventually consistent reads from it. Note about backwards compatibility: In order for the feature to work correctly in presence of log truncations all log replicas need to be updated.</li> +</ul> + + <h2>Upgrading from 1.3.x to 1.4.x</h2> <p><a name="1-4-x-ambient-capabilities"></a></p> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/d81c19af/content/documentation/replicated-log-internals/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/replicated-log-internals/index.html b/content/documentation/replicated-log-internals/index.html index f83f483..dd3d4b2 100644 --- a/content/documentation/replicated-log-internals/index.html +++ b/content/documentation/replicated-log-internals/index.html @@ -143,7 +143,7 @@ <h3>Reaching consensus for a single log entry</h3> -<p>A Paxos round can help all replicas reach consensus on a single log entry’s value. It has two phases: a promise phase and a write phase. Note that we are using slightly different terminology from the <a href="https://research.microsoft.com/en-us/um/people/lamport/pubs/paxes-simple.pdf";>original Paxos paper</a>. In our implementation, the <em>prepare</em> and <em>accept</em> phases in the original paper are referred to as the <em>promise</em> and <em>write</em> phases, respectively. Consequently, a prepare request (response) is referred to as a promise request (response), and an accept request (response) is referred to as a write request (response).</p> +<p>A Paxos round can help all replicas reach consensus on a single log entry’s value. It has two phases: a promise phase and a write phase. Note that we are using slightly different terminology from the <a href="https://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf";>original Paxos paper</a>. In our implementation, the <em>prepare</em> and <em>accept</em> phases in the original paper are referred to as the <em>promise</em> and <em>write</em> phases, respectively. Consequently, a prepare request (response) is referred to as a promise request (response), and an accept request (response) is referred to as a write request (response).</p> <p>To append value <em>X</em> to the log at position <em>p</em>, the coordinator first broadcasts a promise request to all replicas with proposal number <em>n</em>, asking replicas to promise that they will not respond to any request (promise/write request) with a proposal number lower than <em>n</em>. We assume that <em>n</em> is higher than any other previously used proposal number, and will explain how we do this later.</p> @@ -213,12 +213,20 @@ <h3>Auto initialization</h3> -<p>Since we don’t allow an empty replica (a replica in EMPTY status) to respond to requests from coordinators, that raises a question for bootstrapping because initially, each replica is empty. The replicated log provides two choices here. One choice is to use a tool (`mesos-log) to explicitly initialize the log on each replica by setting the replica’s status to VOTING, but that requires an extra step when setting up an application.</p> +<p>Since we don’t allow an empty replica (a replica in EMPTY status) to respond to requests from coordinators, that raises a question for bootstrapping because initially, each replica is empty. The replicated log provides two choices here. One choice is to use a tool (<code>mesos-log</code>) to explicitly initialize the log on each replica by setting the replica’s status to VOTING, but that requires an extra step when setting up an application.</p> <p>The other choice is to do automatic initialization. Our idea is: we allow a replica in EMPTY status to become VOTING immediately if it finds all replicas are in EMPTY status. This is based on the assumption that the only time <em>all</em> replicas are in EMPTY status is during start-up. This may not be true if a catastrophic failure causes all replicas to lose their durable state, and that’s exactly the reason we allow conservative users to disable auto-initialization.</p> <p>To do auto-initialization, if we use a single-phase protocol and allow a replica to directly transit from EMPTY status to VOTING status, we may run into a state where we cannot make progress even if all replicas are in EMPTY status initially. For example, say the quorum size is 2. All replicas are in EMPTY status initially. One replica will first set its status to VOTING because if finds all replicas are in EMPTY status. After that, neither the VOTING replica nor the EMPTY replicas can make progress. To solve this problem, we use a two-phase protocol and introduce an intermediate transient status (STARTING) between EMPTY and VOTING status. A replica in EMPTY status can transit to STARTING status if it finds all replicas are in either EMPTY or STARTING status. A replica in STARTING status can transit to VOTING status if it finds all replicas are in either STARTING or VOTING status. In that way, in our previous example, all replicas will be in STARTING status before any of them can transit to VOTING status.</p> +<h2>Non-leading VOTING replica catch-up</h2> + +<p>Starting with Mesos 1.5.0 it is possible to perform eventually consistent reads from a non-leading VOTING log replica. This makes possible to do additional work on non-leading framework replicas, e.g. offload some reading from a leader to standbys reduce failover time by keeping in-memory storage represented by the replicated log “hot”.</p> + +<p>To serve eventually consistent reads a replica needs to perform <em>catch-up</em> to recover the latest log state in a manner similar to how it is done during <a href="#catch-up">EMPTY replica recovery</a>. After that the recovered positions can be replayed without fear of seeing “holes”.</p> + +<p>A truncation can take place during the non-leading replica catch-up. The replica may try to fill the truncated position if truncation happens after the replica has recovered <em>begin</em> and <em>end</em> positions, which may lead to producing inconsistent data during log replay. In order to protect against it we use a special tombstone flag that signals to the replica that the position was truncated and <em>begin</em> needs to be adjusted. The replica is not blocked from truncations during or after catching-up, which means that the user may need to retry the catch-up procedure if positions that were recovered became truncated during log replay.</p> + <h2>Future work</h2> <p>Currently, replicated log does not support dynamic quorum size change, also known as <em>reconfiguration</em>. Supporting reconfiguration would allow us more easily to add, move or swap hosts for replicas. We plan to support reconfiguration in the future.</p> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/d81c19af/content/documentation/upgrades/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/upgrades/index.html b/content/documentation/upgrades/index.html index 02839af..de8beee 100644 --- a/content/documentation/upgrades/index.html +++ b/content/documentation/upgrades/index.html @@ -546,6 +546,13 @@ changed. Note that if this feature is used, the master version is required to be </ul> +<p><a name="1-5-x-log-reader-catchup"></a></p> + +<ul> +<li>A new <code>catchup()</code> method has been added to the replicated log reader API. The method allows to catch-up positions missing in the local non-leading replica to allow safe eventually consistent reads from it. Note about backwards compatibility: In order for the feature to work correctly in presence of log truncations all log replicas need to be updated.</li> +</ul> + + <h2>Upgrading from 1.3.x to 1.4.x</h2> <p><a name="1-4-x-ambient-capabilities"></a></p>
