Repository: mesos-site Updated Branches: refs/heads/asf-site 494164ba6 -> cc574cb66
Updated the website built from mesos SHA: 52b5899. Project: http://git-wip-us.apache.org/repos/asf/mesos-site/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos-site/commit/cc574cb6 Tree: http://git-wip-us.apache.org/repos/asf/mesos-site/tree/cc574cb6 Diff: http://git-wip-us.apache.org/repos/asf/mesos-site/diff/cc574cb6 Branch: refs/heads/asf-site Commit: cc574cb66debce133d4c5874e7bd17529445697d Parents: 494164b Author: jenkins <bui...@apache.org> Authored: Thu Feb 8 18:40:44 2018 +0000 Committer: jenkins <bui...@apache.org> Committed: Thu Feb 8 18:40:44 2018 +0000 ---------------------------------------------------------------------- .../1.3-1.5_v1_json_state_query_latency.png | Bin 0 -> 19517 bytes content/blog/feed.xml | 131 +++++++- content/blog/index.html | 5 + content/blog/mesos-1-5-0-released/index.html | 316 +++++++++++++++++++ .../index.html | 2 +- content/sitemap.xml | 4 + 6 files changed, 455 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/mesos-site/blob/cc574cb6/content/assets/img/documentation/1.3-1.5_v1_json_state_query_latency.png ---------------------------------------------------------------------- diff --git a/content/assets/img/documentation/1.3-1.5_v1_json_state_query_latency.png b/content/assets/img/documentation/1.3-1.5_v1_json_state_query_latency.png new file mode 100644 index 0000000..fc1d5d9 Binary files /dev/null and b/content/assets/img/documentation/1.3-1.5_v1_json_state_query_latency.png differ http://git-wip-us.apache.org/repos/asf/mesos-site/blob/cc574cb6/content/blog/feed.xml ---------------------------------------------------------------------- diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 77315cc..6dfce75 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -4,7 +4,134 @@ <id>http://mesos.apache.org/blog</id> <link href="http://mesos.apache.org/blog"; /> <link href="http://mesos.apache.org/blog/feed.xml"; rel="self"/> - <updated>2017-12-12T00:00:00Z</updated> + <updated>2018-02-08T00:00:00Z</updated> + + <entry> + <id>http://mesos.apache.org/blog/mesos-1-5-0-released/</id> + <link href="/blog/mesos-1-5-0-released/" /> + <title> + Mesos 1.5: Storage, Performance, Resource Management, and Containerization improvements + </title> + <updated>2018-02-08T00:00:00Z</updated> + <author> + <name>Gilbert Song</name> + </author> + <content type="html"> + <p>I&rsquo;m pleased to announce that Apache Mesos 1.5.0 was released today!</p> + +<h1>New Features and Improvements</h1> + +<p>Mesos 1.5 includes significant improvements to resource management, storage and containerization. Container Storage Interface (CSI) support is new and experimental in Mesos 1.5, and many of the 1.5 features were added to support CSI. The new ability to reconfigure agents has made the operator experience better. Performance gains have made master failover 80-85% faster, and the v1 API performance has improved significantly. Resource management is more flexible with the new release. Last but not least, image garbage collection and better Windows support make it easier to run Mesos on different types of infrastructure, and to run different types of workloads on Mesos.</p> + +<p>Read about all these improvements below, and enjoy Mesos 1.5.</p> + +<h2>Container Storage Interface Support</h2> + +<p>Mesos 1.5 adds experimental support for the <a href="https://github.com/container-storage-interface/spec">Container Storage Interface (CSI)</a>.CSI is a specification that defines a common set of APIs for all interactions between storage vendors and container orchestration platforms. It is the result of a close collaboration among representatives from the Mesos, <a href="https://kubernetes.io/">Kubernetes</a>, <a href="https://www.cloudfoundry.org/">Cloud Foundry</a>, and <a href="https://www.docker.com/">Docker</a> communities. The primary goal of CSI is to allow storage vendors to write one plugin that works with all container orchestration platforms.</p> + +<p>Supporting CSI will help Mesos consistently fit into the larger storage ecosystem, resulting in better storage support in Mesos. Users will be able to use any storage system with Mesos using a consistent API. The out-of-tree plugin model of CSI decouples the release cycle of Mesos from that of the storage systems, making the integration itself more sustainable and maintainable.</p> + +<p><img src="/assets/img/documentation/csi-architecture.png" alt="CSI Support in Mesos" /></p> + +<p>The above figure shows the high-level architecture of how CSI is supported in Mesos. For more details, refer to this <a href="http://mesos.apache.org/documentation/latest/csi/">documentation</a>.</p> + +<h2>Improved Operator Experiences</h2> + +<h3>Agent Reconfiguration Policy</h3> + +<p>Prior to Mesos 1.5, it was not possible to make any changes to the configuration of an agent without killing all tasks running on that agent and restarting with a new agent ID. Any change, for example, an additional attribute or a newly connected hard-drive, would be detected and abort the agent startup with a nasty-looking error:</p> + +<pre><code>EXIT with status 1: Failed to perform recovery: Incompatible agent info detected. +------------------------------------------------------------ +[...] +------------------------------------------------------------ +To remedy this do as follows: +Step 1: rm -f /path/to/work_dir/meta/slaves/latest + This ensures agent doesn't recover old live executors. +Step 2: Restart the agent. +</code></pre> + +<p>Starting with 1.5, operators can use the new agent command-line flag <code>--reconfiguration_policy</code> to configure the types of operations that should be allowed on agents, and which should lead to errors. When setting the value of this flag to <code>additive</code>, the agent will tolerate any changes that increase the amount of resources on the agent, as well as changes that add additional attributes or fault domains.</p> + +<p>In the future, we hope to provide the additional setting <code>--reconfiguration_policy=any</code> to allow any changes to the agent configuration.</p> + +<h2>Performance improvements</h2> + +<p>This release includes improvements to master failover and v1 operator state query API performance. Master failover time-to-completion achieved a 450-600% improvement in throughput, which reduces the time-to-completion by 80-85%.</p> + +<p><img src="/assets/img/documentation/1.3-1.5_master_failover_with_history.png" alt="1.3 - 1.5 Master Failover with Task History Graph" /></p> + +<p>See <a href="http://mesos.apache.org/blog/performance-working-group-progress-report/">our blog post</a> for more details.</p> + +<p>In addition, the performance of v1 operator state query API has also been greatly improved thanks to unnecessary copy eliminations. Specifically, v1 protobuf GetState call is 36% faster compared to v0. And, v1 JSON GetState performance has improved by more than four times.</p> + +<p><img src="/assets/img/documentation/1.3-1.5_v1_json_state_query_latency.png" alt="1.3 - 1.5 v1 JSON State Query Latency" /></p> + +<p>Detailed benchmark results can be found in <a href="https://docs.google.com/spreadsheets/d/1mN4OLEi7UdbLxv-Q3k3iMlmO6Gb2wx9X-gHRnDENSK0/edit#gid=1562845084">this spreadsheet</a>.</p> + +<h2>Resource Management</h2> + +<h3>Quota guarantee improvements</h3> + +<p>Several quota related improvements were made in this release. Now Mesos does a better job ensuring that a role receives its quota and that a role does not exceed its quota, e.g.:</p> + +<ul> +<li>Previously a role could âgameâ the quota system by amassing reservations that it leaves unused. This is now prevented by <a href="https://issues.apache.org/jira/browse/MESOS-4527">accounting for reservations</a> when allocating resources.</li> +<li>Resources are now <a href="https://issues.apache.org/jira/browse/MESOS-7099">allocated in a fine-grained manner</a> to prevent roles from exceeding their quota.</li> +<li>There was a <a href="https://issues.apache.org/jira/browse/MESOS-8293">bug</a> where roles without quota may not receive their reservations. This has been fixed.</li> +<li>When a role had more reservations than its quota, there was a <a href="https://issues.apache.org/jira/browse/MESOS-8339">bug</a> where an insufficient amount of quota headroom was held. This has been fixed.</li> +<li>When allocating resources to a role with quota, we previously also included resources that the role did not have the quota for on the agent. This made it possible to violate the quota guarantees of a different role. This has been fixed by <a href="https://issues.apache.org/jira/browse/MESOS-8352">taking into account the headroom</a> that is needed when allocating the resources.</li> +</ul> + + +<h3>Resource Provider abstraction</h3> + +<p>To decouple resource and agent life cycles we have introduced an abstraction for <a href="https://issues.apache.org/jira/browse/MESOS-7235">resource providers</a>. This lays the groundwork to support dynamic changes to agent resources, integration of external resource providers, or management of cluster-wide resources. As part of the decoupling of resource responsibilities, we have introduced a new protocol to distribute operations on resources. This allows them to explicitly communicate success or failure, and enables reconciliation between cluster components. We have used the new resource provider abstraction to integrate agent-local storage resources interfaced with CSI.</p> + +<h2>Containerization and Multi-Platform Support</h2> + +<h3>Windows Support Improvements</h3> + +<p>With the release of Mesos 1.5, Windows support has been vastly improved. Now when running tasks with the Mesos containerizer, resource limits can be enforced at the operating system level via job objects (<a href="https://issues-test.apache.org/jira/browse/MESOS-6690">MESOS-6690</a>). When the <a href="https://mesos.apache.org/documentation/latest/isolators/windows/#cpu-limits">windows/cpu</a> and <a href="https://mesos.apache.org/documentation/latest/isolators/windows/#memory-limits">windows/mem</a> isolators are used, a hard cap on the CPU cycles and virtual memory can be enforced. Additionally, the CPU and memory statistics are now correctly supported. The Mesos fetcher has been <a href="https://issues-test.apache.org/jira/browse/MESOS-8180">ported to Windows</a>, with support for automatically downloading files over TLS, and extracting ZIP archives. When configured for it, libprocess can now be built with <a href="https://issues-test.apache.org/jira/browse/MESOS-7992">OpenSSL support on Windows</a>, allowing TLS connections to be enforced within the cluster (note that this relies on building or installing OpenSSL externally, as it is not shipped with Windows). The first authentication method, CRAM-MD5, has been enabled on Windows, so that agents can authenticate with masters. Both HTTP and TCP <a href="https://issues-test.apache.org/jira/browse/MESOS-6709">health checks</a> are now available for tasks on Windows agents (for shell tasks at least; health checks for Docker containers are coming soon). The Mesos agent no longer has to be run as an administrator, as it is now possible to <a href="https://issues-test.apache.org/jira/browse/MESOS-7370">create symlinks as a normal user</a>, and this feature was integrated into the agent. Finally, while this was available in the previous release, <a hr ef="https://issues-test.apache.org/jira/browse/MESOS-7371">native long path support for Windows</a> has been implemented, so that running the agent on Windows is now straightforward (with no more registry configurations).</p> + +<h3>Container Image GC</h3> + +<p>Mesos 1.5 has support for Container Image Garbage Collection. There are two ways to garbage collect unused image layers: automatically or manually. This feature helps users avoid unbounded disk space usage in the docker image store. Please see <a href="http://mesos.apache.org/documentation/latest/container-image/#garbage-collect-unused-container-images">Container Image GC Document</a> for more details.</p> + +<h3>Standalone Container</h3> + +<p>This release includes a set of new operator APIs for launching and managing a new primitive called a &ldquo;Standalone Container&rdquo;. A Standalone Container is similar to a container launched by a framework on the Mesos Agent, except that Standalone Containers are launched directly on the Mesos Agent by the operator. As a result, these containers do not use a Mesos Executor and have some other limitations (such as not being able to use reserved resources). At the same time, Standalone Containers are isolated like any other container and can use the same set of containerization features, like container images.</p> + +<p>This feature is being released to support the <a href="http://mesos.apache.org/documentation/latest/csi/">CSI</a> workflow but the APIs are not restricted to this use case. This feature may potentially be used to implement <a href="https://issues.apache.org/jira/browse/MESOS-6595">daemon processes for agent</a> nodes in future. See <a href="https://issues.apache.org/jira/browse/MESOS-7302">MESOS-7302</a> for more details.</p> + +<h2>Mesos Internals</h2> + +<h3>Support gRPC Client</h3> + +<p>This release includes a gRPC client wrapper. <a href="https://grpc.io/">gRPC</a> is an RPC mechanism that is becoming increasingly popular in the cloud-native space as an alternative to REST API. As a first step to integrating gRPC into Mesos, we have added this client wrapper in libprocess to support CSI. This feature is disabled by default, and can be enabled with the <code>--enable-grpc</code> (autotools) build flag on clusters that have been built with the <a href="https://www.openssl.org/">OpenSSL</a> library installed. The wrapper class maintains gRPC runtime data structures internally, and provides a simple interface to make asynchronous gRPC calls that are compatible to libprocessâ actor-based model. We have used this new feature to interface with CSI plugins. In addition, people can also use this new feature to create Mesos modules to talk to their own gRPC-based services.</p> + +<h3>Replicated log improvements</h3> + +<p>Mesos replicated log is used by frameworks (e.g. Apache Aurora) to persist their state across multiple replicas so it can survive failovers and restarts. Starting with 1.5 it is possible to perform eventually consistent reads from a non-leading VOTING replicated log replica. This makes possible to do additional work on non-leading framework replicas, such as offloading some reading from a leader to standbys, or reducing failover time by keeping in-memory storage represented by the replicated log &ldquo;hot&rdquo;.</p> + +<h1>Upgrade</h1> + +<p>Rolling upgrades from a Mesos 1.4.0 cluster to Mesos 1.5.0 are straightforward. Please refer to the <a href="http://mesos.apache.org/documentation/latest/upgrades/">upgrade guide</a> for detailed information on upgrading to Mesos 1.5.0.</p> + +<h1>Community</h1> + +<p>Inspired by the work that went into this release? Want to get involved? Have feedback? We&rsquo;d love to hear from you! Join a <a href="http://mesos.apache.org/documentation/latest/working-groups/">working group</a> or start a conversation in the <a href="http://mesos.apache.org/community/">community</a>!</p> + +<h1>Thank you!</h1> + +<p>Thanks to the 59 contributors who made Mesos 1.5.0 possible:</p> + +<p>Aaron Wood, Adam B, Adam Dangoor, Akash Gupta, Alastair Montgomery, Alexander Rojas, Alexander Rukletsov, Anand Mazumdar, Andrei Budnik, Andrew Schwartzmeyer, Andrey Dudin, Andy Pang, Anindya Sinha, Armand Grillet, Avinash sridharan, Benjamin Bannier, Benjamin Hindman, Benjamin Mahler, Benno Evers, Bob Eckert, Chun-Hung Hsiao, Cynthia Thomas, Deepak Goel, Dmitry Zhuk, Gaston Kleiman, Gastón Kleiman, Gilbert Song, Greg Mann, Ilya Pronin, Jack Yang, James DeFelice, James Peach, Jan Schlicht, Jason Lai, Jeff Coffler, Jiang Yan Xu, Jie Yu, Joerg Schad, John Kordich, Joseph Wu, Julien Pepy, Kapil Arya, Kevin Klues, Megha Sharma, Meng Zhu, Michael Park, Nathan Jackson, Neil Conway, Packt, Pranay Kanwar, Qian Zhang, Quinn Leng, Richard Shaw, ThodorisZois, Till Toenshoff, Tomas Barton, Tomasz Janiszewski, Vinod Kone, Zhitao Li</p> + +<p>Enjoy Apache Mesos 1.5!</p> + + </content> + </entry> <entry> <id>http://mesos.apache.org/blog/storage-developments-in-apache-mesos/</id> @@ -168,7 +295,7 @@ To learn more about CSI work in Mesos, you can dig into the design document < </ul> -<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="&#x6d;&#x61;&#x69;&#108;&#x74;&#111;&#x3a;&#x64;&#101;&#x76;&#64;&#97;&#112;&#x61;&#99;&#x68;&#x65;&#46;&#x6d;&#101;&#115;&#111;&#115;&#x2e;&#x6f;&#114;&#x67;">&#100;&#x65;&#x76;&#64;&#97;&#112;&#97;&#99;&#x68;&#101;&#x2e;&#x6d;&#x65;&#115;&#x6f;&#x73;&#46;&#x6f;&#x72;&#x67;</a>.</p> +<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="&#x6d;&#x61;&#x69;&#108;&#x74;&#x6f;&#x3a;&#100;&#x65;&#x76;&#64;&#97;&#x70;&#97;&#99;&#104;&#x65;&#46;&#x6d;&#101;&#x73;&#x6f;&#x73;&#46;&#x6f;&#x72;&#x67;">&#x64;&#101;&#118;&#x40;&#x61;&#112;&#97;&#x63;&#x68;&#x65;&#x2e;&#109;&#x65;&#115;&#111;&#x73;&#x2e;&#111;&#x72;&#103;</a>.</p> </content> </entry> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/cc574cb6/content/blog/index.html ---------------------------------------------------------------------- diff --git a/content/blog/index.html b/content/blog/index.html index 109135d..8dc6003 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -107,6 +107,11 @@ <div class="col-md-9"> <article> + <h2><a href="/blog/mesos-1-5-0-released/">Mesos 1.5: Storage, Performance, Resource Management, and Containerization improvements</a></h2> + <p><em>Posted by Gilbert Song, February 8, 2018</em></p> + </article> + + <article> <h2><a href="/blog/storage-developments-in-apache-mesos/">Storage developments in Apache Mesos: Interview with Jie Yu</a></h2> <p><em>Posted by Greg Mann, December 12, 2017</em></p> </article> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/cc574cb6/content/blog/mesos-1-5-0-released/index.html ---------------------------------------------------------------------- diff --git a/content/blog/mesos-1-5-0-released/index.html b/content/blog/mesos-1-5-0-released/index.html new file mode 100644 index 0000000..b2a9ad3 --- /dev/null +++ b/content/blog/mesos-1-5-0-released/index.html @@ -0,0 +1,316 @@ +<!DOCTYPE html> +<html> + <head> + <meta charset="utf-8"> + <title>Mesos 1.5: Storage, Performance, Resource Management, and Containerization improvements</title> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <meta property="og:locale" content="en_US"/> + <meta property="og:type" content="website"/> + <meta property="og:title" content="Apache Mesos"/> + <meta property="og:site_name" content="Apache Mesos"/> + <meta property="og:url" content="http://mesos.apache.org/"/> + <meta property="og:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/> + <meta property="og:description" + content="Apache Mesos abstracts resources away from machines, + enabling fault-tolerant and elastic distributed systems + to easily be built and run effectively."/> + + <meta name="twitter:card" content="summary"/> + <meta name="twitter:site" content="@ApacheMesos"/> + <meta name="twitter:title" content="Apache Mesos"/> + <meta name="twitter:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/> + <meta name="twitter:description" + content="Apache Mesos abstracts resources away from machines, + enabling fault-tolerant and elastic distributed systems + to easily be built and run effectively."/> + + <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet"> + <link rel="alternate" type="application/atom+xml" title="Apache Mesos Blog" href="/blog/feed.xml"> + <link href="../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" /> + + + + <!-- Google Analytics Magic --> + <script type="text/javascript"> + var _gaq = _gaq || []; + _gaq.push(['_setAccount', 'UA-20226872-1']); + _gaq.push(['_setDomainName', 'apache.org']); + _gaq.push(['_trackPageview']); + + (function() { + var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; + ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); + })(); + </script> + + </head> + <body> + <!-- magical breadcrumbs --> + <div class="topnav"> + <div class="container"> + <ul class="breadcrumb"> + <li> + <div class="dropdown"> + <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a> + <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel"> + <li><a href="http://www.apache.org";>Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + </ul> + </div> + </li> + + <li><a href="http://mesos.apache.org";>Apache Mesos</a></li> + + + <li><a href="/blog +/">Blog +</a></li> + + + </ul><!-- /.breadcrumb --> + </div><!-- /.container --> + </div><!-- /.topnav --> + + <!-- navbar excitement --> +<div class="navbar navbar-default navbar-static-top" role="navigation"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#mesos-menu" aria-expanded="false"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo"/></a> + </div><!-- /.navbar-header --> + + <div class="navbar-collapse collapse" id="mesos-menu"> + <ul class="nav navbar-nav navbar-right"> + <li><a href="/getting-started/">Getting Started</a></li> + <li><a href="/blog/">Blog</a></li> + <li><a href="/documentation/latest/">Documentation</a></li> + <li><a href="/downloads/">Downloads</a></li> + <li><a href="/community/">Community</a></li> + </ul> + </div><!-- /#mesos-menu --> + </div><!-- /.container --> +</div><!-- /.navbar --> + +<div class="content"> + <div class="container"> + <div class="row"> + + <div class="col-md-3"> + <div class="meta"> + <span class="author"> + + <img src="http://www.gravatar.com/avatar/05d3596cf7ef7751c02545b1eaac64ac?s=80"; + class="author_gravatar"> + + <span class="author_contact"> + <p><strong>Gilbert Song</strong></p> + + <p><a href="http://twitter.com/Gilbert_Songs";>@Gilbert_Songs</a></p> + + </span> + </span> + <p><em>Posted February 8, 2018</em></p> + </div> + + <div class="share"> + <span class="social-share-button"><a href="https://twitter.com/share"; class="twitter-share-button" data-via="apachemesos">Tweet</a></span> + + <span><script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script></span> + + <span><div class="g-plusone" data-size="medium"></div></span> + + <!-- Place this tag after the last +1 button tag. --> + <script type="text/javascript"> + (function() { + var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; + po.src = 'https://apis.google.com/js/plusone.js'; + var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); + })(); + </script> + + <script src="//platform.linkedin.com/in.js" type="text/javascript"> + lang: en_US + </script> + <script type="IN/Share" data-counter="right"></script> + </div> + </div> + + <div class="post col-md-9"> + <h1>Mesos 1.5: Storage, Performance, Resource Management, and Containerization improvements</h1> + + <p>I’m pleased to announce that Apache Mesos 1.5.0 was released today!</p> + +<h1>New Features and Improvements</h1> + +<p>Mesos 1.5 includes significant improvements to resource management, storage and containerization. Container Storage Interface (CSI) support is new and experimental in Mesos 1.5, and many of the 1.5 features were added to support CSI. The new ability to reconfigure agents has made the operator experience better. Performance gains have made master failover 80-85% faster, and the v1 API performance has improved significantly. Resource management is more flexible with the new release. Last but not least, image garbage collection and better Windows support make it easier to run Mesos on different types of infrastructure, and to run different types of workloads on Mesos.</p> + +<p>Read about all these improvements below, and enjoy Mesos 1.5.</p> + +<h2>Container Storage Interface Support</h2> + +<p>Mesos 1.5 adds experimental support for the <a href="https://github.com/container-storage-interface/spec";>Container Storage Interface (CSI)</a>.CSI is a specification that defines a common set of APIs for all interactions between storage vendors and container orchestration platforms. It is the result of a close collaboration among representatives from the Mesos, <a href="https://kubernetes.io/";>Kubernetes</a>, <a href="https://www.cloudfoundry.org/";>Cloud Foundry</a>, and <a href="https://www.docker.com/";>Docker</a> communities. The primary goal of CSI is to allow storage vendors to write one plugin that works with all container orchestration platforms.</p> + +<p>Supporting CSI will help Mesos consistently fit into the larger storage ecosystem, resulting in better storage support in Mesos. Users will be able to use any storage system with Mesos using a consistent API. The out-of-tree plugin model of CSI decouples the release cycle of Mesos from that of the storage systems, making the integration itself more sustainable and maintainable.</p> + +<p><img src="/assets/img/documentation/csi-architecture.png" alt="CSI Support in Mesos" /></p> + +<p>The above figure shows the high-level architecture of how CSI is supported in Mesos. For more details, refer to this <a href="http://mesos.apache.org/documentation/latest/csi/";>documentation</a>.</p> + +<h2>Improved Operator Experiences</h2> + +<h3>Agent Reconfiguration Policy</h3> + +<p>Prior to Mesos 1.5, it was not possible to make any changes to the configuration of an agent without killing all tasks running on that agent and restarting with a new agent ID. Any change, for example, an additional attribute or a newly connected hard-drive, would be detected and abort the agent startup with a nasty-looking error:</p> + +<pre><code>EXIT with status 1: Failed to perform recovery: Incompatible agent info detected. +------------------------------------------------------------ +[...] +------------------------------------------------------------ +To remedy this do as follows: +Step 1: rm -f /path/to/work_dir/meta/slaves/latest + This ensures agent doesn't recover old live executors. +Step 2: Restart the agent. +</code></pre> + +<p>Starting with 1.5, operators can use the new agent command-line flag <code>--reconfiguration_policy</code> to configure the types of operations that should be allowed on agents, and which should lead to errors. When setting the value of this flag to <code>additive</code>, the agent will tolerate any changes that increase the amount of resources on the agent, as well as changes that add additional attributes or fault domains.</p> + +<p>In the future, we hope to provide the additional setting <code>--reconfiguration_policy=any</code> to allow any changes to the agent configuration.</p> + +<h2>Performance improvements</h2> + +<p>This release includes improvements to master failover and v1 operator state query API performance. Master failover time-to-completion achieved a 450-600% improvement in throughput, which reduces the time-to-completion by 80-85%.</p> + +<p><img src="/assets/img/documentation/1.3-1.5_master_failover_with_history.png" alt="1.3 - 1.5 Master Failover with Task History Graph" /></p> + +<p>See <a href="http://mesos.apache.org/blog/performance-working-group-progress-report/";>our blog post</a> for more details.</p> + +<p>In addition, the performance of v1 operator state query API has also been greatly improved thanks to unnecessary copy eliminations. Specifically, v1 protobuf GetState call is 36% faster compared to v0. And, v1 JSON GetState performance has improved by more than four times.</p> + +<p><img src="/assets/img/documentation/1.3-1.5_v1_json_state_query_latency.png" alt="1.3 - 1.5 v1 JSON State Query Latency" /></p> + +<p>Detailed benchmark results can be found in <a href="https://docs.google.com/spreadsheets/d/1mN4OLEi7UdbLxv-Q3k3iMlmO6Gb2wx9X-gHRnDENSK0/edit#gid=1562845084";>this spreadsheet</a>.</p> + +<h2>Resource Management</h2> + +<h3>Quota guarantee improvements</h3> + +<p>Several quota related improvements were made in this release. Now Mesos does a better job ensuring that a role receives its quota and that a role does not exceed its quota, e.g.:</p> + +<ul> +<li>Previously a role could âgameâ the quota system by amassing reservations that it leaves unused. This is now prevented by <a href="https://issues.apache.org/jira/browse/MESOS-4527";>accounting for reservations</a> when allocating resources.</li> +<li>Resources are now <a href="https://issues.apache.org/jira/browse/MESOS-7099";>allocated in a fine-grained manner</a> to prevent roles from exceeding their quota.</li> +<li>There was a <a href="https://issues.apache.org/jira/browse/MESOS-8293";>bug</a> where roles without quota may not receive their reservations. This has been fixed.</li> +<li>When a role had more reservations than its quota, there was a <a href="https://issues.apache.org/jira/browse/MESOS-8339";>bug</a> where an insufficient amount of quota headroom was held. This has been fixed.</li> +<li>When allocating resources to a role with quota, we previously also included resources that the role did not have the quota for on the agent. This made it possible to violate the quota guarantees of a different role. This has been fixed by <a href="https://issues.apache.org/jira/browse/MESOS-8352";>taking into account the headroom</a> that is needed when allocating the resources.</li> +</ul> + + +<h3>Resource Provider abstraction</h3> + +<p>To decouple resource and agent life cycles we have introduced an abstraction for <a href="https://issues.apache.org/jira/browse/MESOS-7235";>resource providers</a>. This lays the groundwork to support dynamic changes to agent resources, integration of external resource providers, or management of cluster-wide resources. As part of the decoupling of resource responsibilities, we have introduced a new protocol to distribute operations on resources. This allows them to explicitly communicate success or failure, and enables reconciliation between cluster components. We have used the new resource provider abstraction to integrate agent-local storage resources interfaced with CSI.</p> + +<h2>Containerization and Multi-Platform Support</h2> + +<h3>Windows Support Improvements</h3> + +<p>With the release of Mesos 1.5, Windows support has been vastly improved. Now when running tasks with the Mesos containerizer, resource limits can be enforced at the operating system level via job objects (<a href="https://issues-test.apache.org/jira/browse/MESOS-6690";>MESOS-6690</a>). When the <a href="https://mesos.apache.org/documentation/latest/isolators/windows/#cpu-limits";>windows/cpu</a> and <a href="https://mesos.apache.org/documentation/latest/isolators/windows/#memory-limits";>windows/mem</a> isolators are used, a hard cap on the CPU cycles and virtual memory can be enforced. Additionally, the CPU and memory statistics are now correctly supported. The Mesos fetcher has been <a href="https://issues-test.apache.org/jira/browse/MESOS-8180";>ported to Windows</a>, with support for automatically downloading files over TLS, and extracting ZIP archives. When configured for it, libprocess can now be built with <a href="https://issues-test.apache.org/jira/browse/MESOS-7992";>OpenSSL support on Windows</a>, allowing TLS connections to be enforced within the cluster (note that this relies on building or installing OpenSSL externally, as it is not shipped with Windows). The first authentication method, CRAM-MD5, has been enabled on Windows, so that agents can authenticate with masters. Both HTTP and TCP <a href="https://issues-test.apache.org/jira/browse/MESOS-6709";>health checks</a> are now available for tasks on Windows agents (for shell tasks at least; health checks for Docker containers are coming soon). The Mesos agent no longer has to be run as an administrator, as it is now possible to <a href="https://issues-test.apache.org/jira/browse/MESOS-7370";>create symlinks as a normal user</a>, and this feature was integrated into the agent. Finally, while this was available in the previous release, <a href="https://issues-test.apache.org/jira/browse/MESOS-7371";>native long path support for Windows</a> has been implemented, so that running the agent on Windows is n ow straightforward (with no more registry configurations).</p> + +<h3>Container Image GC</h3> + +<p>Mesos 1.5 has support for Container Image Garbage Collection. There are two ways to garbage collect unused image layers: automatically or manually. This feature helps users avoid unbounded disk space usage in the docker image store. Please see <a href="http://mesos.apache.org/documentation/latest/container-image/#garbage-collect-unused-container-images";>Container Image GC Document</a> for more details.</p> + +<h3>Standalone Container</h3> + +<p>This release includes a set of new operator APIs for launching and managing a new primitive called a “Standalone Container”. A Standalone Container is similar to a container launched by a framework on the Mesos Agent, except that Standalone Containers are launched directly on the Mesos Agent by the operator. As a result, these containers do not use a Mesos Executor and have some other limitations (such as not being able to use reserved resources). At the same time, Standalone Containers are isolated like any other container and can use the same set of containerization features, like container images.</p> + +<p>This feature is being released to support the <a href="http://mesos.apache.org/documentation/latest/csi/";>CSI</a> workflow but the APIs are not restricted to this use case. This feature may potentially be used to implement <a href="https://issues.apache.org/jira/browse/MESOS-6595";>daemon processes for agent</a> nodes in future. See <a href="https://issues.apache.org/jira/browse/MESOS-7302";>MESOS-7302</a> for more details.</p> + +<h2>Mesos Internals</h2> + +<h3>Support gRPC Client</h3> + +<p>This release includes a gRPC client wrapper. <a href="https://grpc.io/";>gRPC</a> is an RPC mechanism that is becoming increasingly popular in the cloud-native space as an alternative to REST API. As a first step to integrating gRPC into Mesos, we have added this client wrapper in libprocess to support CSI. This feature is disabled by default, and can be enabled with the <code>--enable-grpc</code> (autotools) build flag on clusters that have been built with the <a href="https://www.openssl.org/";>OpenSSL</a> library installed. The wrapper class maintains gRPC runtime data structures internally, and provides a simple interface to make asynchronous gRPC calls that are compatible to libprocessâ actor-based model. We have used this new feature to interface with CSI plugins. In addition, people can also use this new feature to create Mesos modules to talk to their own gRPC-based services.</p> + +<h3>Replicated log improvements</h3> + +<p>Mesos replicated log is used by frameworks (e.g. Apache Aurora) to persist their state across multiple replicas so it can survive failovers and restarts. Starting with 1.5 it is possible to perform eventually consistent reads from a non-leading VOTING replicated log replica. This makes possible to do additional work on non-leading framework replicas, such as offloading some reading from a leader to standbys, or reducing failover time by keeping in-memory storage represented by the replicated log “hot”.</p> + +<h1>Upgrade</h1> + +<p>Rolling upgrades from a Mesos 1.4.0 cluster to Mesos 1.5.0 are straightforward. Please refer to the <a href="http://mesos.apache.org/documentation/latest/upgrades/";>upgrade guide</a> for detailed information on upgrading to Mesos 1.5.0.</p> + +<h1>Community</h1> + +<p>Inspired by the work that went into this release? Want to get involved? Have feedback? We’d love to hear from you! Join a <a href="http://mesos.apache.org/documentation/latest/working-groups/";>working group</a> or start a conversation in the <a href="http://mesos.apache.org/community/";>community</a>!</p> + +<h1>Thank you!</h1> + +<p>Thanks to the 59 contributors who made Mesos 1.5.0 possible:</p> + +<p>Aaron Wood, Adam B, Adam Dangoor, Akash Gupta, Alastair Montgomery, Alexander Rojas, Alexander Rukletsov, Anand Mazumdar, Andrei Budnik, Andrew Schwartzmeyer, Andrey Dudin, Andy Pang, Anindya Sinha, Armand Grillet, Avinash sridharan, Benjamin Bannier, Benjamin Hindman, Benjamin Mahler, Benno Evers, Bob Eckert, Chun-Hung Hsiao, Cynthia Thomas, Deepak Goel, Dmitry Zhuk, Gaston Kleiman, Gastón Kleiman, Gilbert Song, Greg Mann, Ilya Pronin, Jack Yang, James DeFelice, James Peach, Jan Schlicht, Jason Lai, Jeff Coffler, Jiang Yan Xu, Jie Yu, Joerg Schad, John Kordich, Joseph Wu, Julien Pepy, Kapil Arya, Kevin Klues, Megha Sharma, Meng Zhu, Michael Park, Nathan Jackson, Neil Conway, Packt, Pranay Kanwar, Qian Zhang, Quinn Leng, Richard Shaw, ThodorisZois, Till Toenshoff, Tomas Barton, Tomasz Janiszewski, Vinod Kone, Zhitao Li</p> + +<p>Enjoy Apache Mesos 1.5!</p> + + </div> +</div> + + </div><!-- /.container --> +</div><!-- /.content --> + +<hr> + + + + <!-- footer --> + <div class="footer"> + <div class="container"> + <div class="col-md-4 social-blk"> + <span class="social"> + <a href="https://twitter.com/ApacheMesos"; + class="twitter-follow-button" + data-show-count="false" data-size="large">Follow @ApacheMesos</a> + <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> + <a href="https://twitter.com/intent/tweet?button_hashtag=mesos"; + class="twitter-hashtag-button" + data-size="large" + data-related="ApacheMesos">Tweet #mesos</a> + <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> + </span> + </div> + + <div class="col-md-8 trademark"> + <p>© 2012-2017 <a href="http://apache.org";>The Apache Software Foundation</a>. + Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation. + <p> + </div> + </div><!-- /.container --> + </div><!-- /.footer --> + + <!-- JS --> + <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script> + <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script> + <script src="//cdnjs.cloudflare.com/ajax/libs/anchor-js/4.1.0/anchor.min.js" type="text/javascript"></script> + + <!-- Inject anchors for all headings on the page, see https://www.bryanbraun.com/anchorjs. --> + <script type="text/javascript"> + anchors.options = { + placement: 'right', + ariaLabel: 'Permalink', + }; + + // The default is to not add anchors to h1, but we have pages with multiple h1 headers, + // and we do want to put anchors on those. + anchors.add('h1, h2, h3, h4, h5, h6'); + </script> + </body> +</html> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/cc574cb6/content/blog/performance-working-group-progress-report/index.html ---------------------------------------------------------------------- diff --git a/content/blog/performance-working-group-progress-report/index.html b/content/blog/performance-working-group-progress-report/index.html index 16cab1c..bc2a032 100644 --- a/content/blog/performance-working-group-progress-report/index.html +++ b/content/blog/performance-working-group-progress-report/index.html @@ -248,7 +248,7 @@ </ul> -<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="mailto:dev@apache.mesos.org">dev@apache.mesos.org</a>.</p> +<p>If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing <a href="mailto:dev@apache.mesos.org">dev@apache.mesos.org</a>.</p> </div> </div> http://git-wip-us.apache.org/repos/asf/mesos-site/blob/cc574cb6/content/sitemap.xml ---------------------------------------------------------------------- diff --git a/content/sitemap.xml b/content/sitemap.xml index e33cabc..a375b41 100644 --- a/content/sitemap.xml +++ b/content/sitemap.xml @@ -17509,6 +17509,10 @@ <lastmod>2018-02-08T00:00:00+00:00</lastmod> </url> <url> + <loc>http://mesos.apache.org/blog/mesos-1-5-0-released/</loc> + <lastmod>2018-02-08T00:00:00+00:00</lastmod> + </url> + <url> <loc>http://mesos.apache.org/blog/mesos-1-1-3-released/</loc> <lastmod>2018-02-08T00:00:00+00:00</lastmod> </url>
![http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/](http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/){kind=link}