Added: dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperJMX.html ============================================================================== --- dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperJMX.html (added) +++ dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperJMX.html Mon Jul 17 12:57:54 2023 @@ -0,0 +1,238 @@ + +<!DOCTYPE html> +<html> +<head> + <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>ZooKeeper: Because Coordinating Distributed Systems is a Zoo</title> + <link type="text/css" href="skin/basic.css" rel="stylesheet"> + <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet"> + <link media="print" type="text/css" href="skin/print.css" rel="stylesheet"> + <link type="text/css" href="skin/profile.css" rel="stylesheet"> + <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script> + <script src="skin/getMenu.js" language="javascript" type="text/javascript"></script> + <script src="skin/init.js" language="javascript" type="text/javascript"></script> + <link rel="shortcut icon" href="images/favicon.ico"> +</head> +<body onload="init();"> +<div id="top"> + <div class="breadtrail"> + <a href="http://www.apache.org/";>Apache</a> > <a href="http://zookeeper.apache.org/";>ZooKeeper</a> + </div> + <div class="header"> + <div class="projectlogo"> + <a href="http://zookeeper.apache.org/";><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a> + </div> + <div class="searchbox"> + <form action="http://www.google.com/search"; method="get"> + <input value="zookeeper.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google"> + <input name="Search" value="Search" type="submit"> + </form> + </div> + <ul id="tabs"> + <li> + <a class="unselected" href="http://zookeeper.apache.org/";>Project</a> + </li> + <li> + <a class="unselected" href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/";>Wiki</a> + </li> + <li class="current"> + <a class="selected" href="index.html">ZooKeeper 3.8 Documentation</a> + </li> + </ul> + </div> +</div> +<div id="main"> + <div id="publishedStrip"> + <div id="level2tabs"></div> + <script type="text/javascript"><!-- +document.write("Last Published: " + document.lastModified); +// --></script> + </div> + <div class="breadtrail"> + + </div> + <div id="menu"> + <div onclick="SwitchMenu('menu_1', 'skin/')" id="menu_1Title" class="menutitle">Overview</div> + <div id="menu_1" class="menuitemgroup"> + <div class="menuitem"> + <a href="index.html">Welcome</a> + </div> + <div class="menuitem"> + <a href="zookeeperOver.html">Overview</a> + </div> + <div class="menuitem"> + <a href="zookeeperStarted.html">Getting Started</a> + </div> + <div class="menuitem"> + <a href="releasenotes.html">Release Notes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_2', 'skin/')" id="menu_2Title" class="menutitle">Developer</div> + <div id="menu_2" class="menuitemgroup"> + <div class="menuitem"> + <a href="apidocs/zookeeper-server/index.html">API Docs</a> + </div> + <div class="menuitem"> + <a href="zookeeperProgrammers.html">Programmer's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperUseCases.html">Use Cases</a> + </div> + <div class="menuitem"> + <a href="javaExample.html">Java Example</a> + </div> + <div class="menuitem"> + <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a> + </div> + <div class="menuitem"> + <a href="recipes.html">Recipes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_3', 'skin/')" id="menu_3Title" class="menutitle">Admin & Ops</div> + <div id="menu_3" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperAdmin.html">Administrator's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperQuotas.html">Quota Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperSnapshotAndRestore.html">Snapshot and Restore Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperJMX.html">JMX</a> + </div> + <div class="menuitem"> + <a href="zookeeperHierarchicalQuorums.html">Hierarchical Quorums</a> + </div> + <div class="menuitem"> + <a href="zookeeperOracleQuorums.html">Oracle Quorum</a> + </div> + <div class="menuitem"> + <a href="zookeeperObservers.html">Observers Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperReconfig.html">Dynamic Reconfiguration</a> + </div> + <div class="menuitem"> + <a href="zookeeperCLI.html">ZooKeeper CLI</a> + </div> + <div class="menuitem"> + <a href="zookeeperTools.html">ZooKeeper Tools</a> + </div> + <div class="menuitem"> + <a href="zookeeperMonitor.html">ZooKeeper Monitor</a> + </div> + <div class="menuitem"> + <a href="zookeeperAuditLogs.html">Audit Logs</a> + </div> + </div> + <div onclick="SwitchMenu('menu_4', 'skin/')" id="menu_4Title" class="menutitle">Contributor</div> + <div id="menu_4" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperInternals.html">ZooKeeper Internals</a> + </div> + </div> + <div onclick="SwitchMenu('menu_5', 'skin/')" id="menu_5Title" class="menutitle">Miscellaneous</div> + <div id="menu_5" class="menuitemgroup"> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER";>Wiki</a> + </div> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ";>FAQ</a> + </div> + <div class="menuitem"> + <a href="http://zookeeper.apache.org/mailing_lists.html";>Mailing Lists</a> + </div> + </div> + </div> + <div id="content"> +<!-- +Copyright 2002-2022 The Apache Software Foundation + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +//--> +<h1>ZooKeeper JMX</h1> +<ul> +<li><a href="#ch_jmx">JMX</a></li> +<li><a href="#ch_starting">Starting ZooKeeper with JMX enabled</a></li> +<li><a href="#ch_console">Run a JMX console</a></li> +<li><a href="#ch_reference">ZooKeeper MBean Reference</a></li> +</ul> +<p><a name="ch_jmx"></a></p> +<h2>JMX</h2> +<p>Apache ZooKeeper has extensive support for JMX, allowing you to view and manage a ZooKeeper serving ensemble.</p> +<p>This document assumes that you have basic knowledge of JMX. See <a href="http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/";>Sun JMX Technology</a> page to get started with JMX.</p> +<p>See the <a href="http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html";>JMX Management Guide</a> for details on setting up local and remote management of VM instances. By default the included <em>zkServer.sh</em> supports only local management - review the linked document to enable support for remote management (beyond the scope of this document).</p> +<p><a name="ch_starting"></a></p> +<h2>Starting ZooKeeper with JMX enabled</h2> +<p>The class <em>org.apache.zookeeper.server.quorum.QuorumPeerMain</em> will start a JMX manageable ZooKeeper server. This class registers the proper MBeans during initialization to support JMX monitoring and management of the instance. See <em>bin/zkServer.sh</em> for one example of starting ZooKeeper using QuorumPeerMain.</p> +<p><a name="ch_console"></a></p> +<h2>Run a JMX console</h2> +<p>There are a number of JMX consoles available which can connect to the running server. For this example we will use Sun's <em>jconsole</em>.</p> +<p>The Java JDK ships with a simple JMX console named <a href="http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html";>jconsole</a> which can be used to connect to ZooKeeper and inspect a running server. Once you've started ZooKeeper using QuorumPeerMain start <em>jconsole</em>, which typically resides in <em>JDK_HOME/bin/jconsole</em></p> +<p>When the "new connection" window is displayed either connect to local process (if jconsole started on the same host as Server) or use the remote process connection.</p> +<p>By default the "overview" tab for the VM is displayed (this is a great way to get insight into the VM btw). Select the "MBeans" tab.</p> +<p>You should now see <em>org.apache.ZooKeeperService</em> on the left hand side. Expand this item and depending on how you've started the server you will be able to monitor and manage various service related features.</p> +<h4>Logback MBeans registration <em>(new in 3.8.0)</em></h4> +<p>Logback is the default logging backend of ZooKeeper since version 3.8.0. It can be configured to register JMX MBeans by adding <code><jmxConfigurator /></code> to <em>logback.xml</em>. More information can be found on Logback's <a href="https://logback.qos.ch/manual/jmxConfig.html";>website</a>.</p> +<h4>Log4j MBeans <em>(deprecated)</em></h4> +<p>Also note that ZooKeeper will register log4j MBeans as well if log4j1 is configured as the logging backend of SLF4j. In the same section along the left hand side you will see "log4j". Expand that to manage log4j through JMX. Of particular interest is the ability to dynamically change the logging levels used by editing the appender and root thresholds. Log4j MBean registration can be disabled by passing <em>-Dzookeeper.jmx.log4j.disable=true</em> to the JVM when starting ZooKeeper. In addition, we can specify the name of the MBean with the <em>-Dzookeeper.jmx.log4j.mbean=log4j:hierarchy=default</em> option, in case we need to upgrade an integrated system using the old MBean name (<code>log4j:hierarchy = default</code>).</p> +<p><a name="ch_reference"></a></p> +<h2>ZooKeeper MBean Reference</h2> +<p>This table details JMX for a server participating in a replicated ZooKeeper ensemble (ie not standalone). This is the typical case for a production environment.</p> +<h3>MBeans, their names and description</h3> +<table> +<thead> +<tr><th> MBean </th><th> MBean Object Name </th><th> Description </th></tr> +</thead> +<tbody> +<tr><td> Quorum </td><td> ReplicatedServer_id<#> </td><td> Represents the Quorum, or Ensemble - parent of all cluster members. Note that the object name includes the "myid" of the server (name suffix) that your JMX agent has connected to. </td></tr> +<tr><td> LocalPeer/RemotePeer </td><td> replica.<#> </td><td> Represents a local or remote peer (ie server participating in the ensemble). Note that the object name includes the "myid" of the server (name suffix). </td></tr> +<tr><td> LeaderElection </td><td> LeaderElection </td><td> Represents a ZooKeeper cluster leader election which is in progress. Provides information about the election, such as when it started. </td></tr> +<tr><td> Leader </td><td> Leader </td><td> Indicates that the parent replica is the leader and provides attributes/operations for that server. Note that Leader is a subclass of ZooKeeperServer, so it provides all of the information normally associated with a ZooKeeperServer node. </td></tr> +<tr><td> Follower </td><td> Follower </td><td> Indicates that the parent replica is a follower and provides attributes/operations for that server. Note that Follower is a subclass of ZooKeeperServer, so it provides all of the information normally associated with a ZooKeeperServer node. </td></tr> +<tr><td> DataTree </td><td> InMemoryDataTree </td><td> Statistics on the in memory znode database, also operations to access finer (and more computationally intensive) statistics on the data (such as ephemeral count). InMemoryDataTrees are children of ZooKeeperServer nodes. </td></tr> +<tr><td> ServerCnxn </td><td> <session_id> </td><td> Statistics on each client connection, also operations on those connections (such as termination). Note the object name is the session id of the connection in hex form. </td></tr> +</tbody> +</table> +<p>This table details JMX for a standalone server. Typically standalone is only used in development situations.</p> +<h3>MBeans, their names and description</h3> +<table> +<thead> +<tr><th> MBean </th><th> MBean Object Name </th><th> Description </th></tr> +</thead> +<tbody> +<tr><td> ZooKeeperServer </td><td> StandaloneServer_port<#> </td><td> Statistics on the running server, also operations to reset these attributes. Note that the object name includes the client port of the server (name suffix). </td></tr> +<tr><td> DataTree </td><td> InMemoryDataTree </td><td> Statistics on the in memory znode database, also operations to access finer (and more computationally intensive) statistics on the data (such as ephemeral count). </td></tr> +<tr><td> ServerCnxn </td><td> < session_id > </td><td> Statistics on each client connection, also operations on those connections (such as termination). Note the object name is the session id of the connection in hex form. </td></tr> +</tbody> +</table> +</div> +<div class="clearboth"> </div> +</div> +<div id="footer"> + <div class="lastmodified"> + <script type="text/javascript"> + <!-- + document.write("Last Published: " + document.lastModified); + // --> + </script> + </div> + <div class="copyright"> + Copyright © <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> + </div> + <div id="logos"></div> +</div> +</body> +</html> \ No newline at end of file
Added: dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperMonitor.html ============================================================================== --- dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperMonitor.html (added) +++ dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperMonitor.html Mon Jul 17 12:57:54 2023 @@ -0,0 +1,380 @@ + +<!DOCTYPE html> +<html> +<head> + <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>ZooKeeper: Because Coordinating Distributed Systems is a Zoo</title> + <link type="text/css" href="skin/basic.css" rel="stylesheet"> + <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet"> + <link media="print" type="text/css" href="skin/print.css" rel="stylesheet"> + <link type="text/css" href="skin/profile.css" rel="stylesheet"> + <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script> + <script src="skin/getMenu.js" language="javascript" type="text/javascript"></script> + <script src="skin/init.js" language="javascript" type="text/javascript"></script> + <link rel="shortcut icon" href="images/favicon.ico"> +</head> +<body onload="init();"> +<div id="top"> + <div class="breadtrail"> + <a href="http://www.apache.org/";>Apache</a> > <a href="http://zookeeper.apache.org/";>ZooKeeper</a> + </div> + <div class="header"> + <div class="projectlogo"> + <a href="http://zookeeper.apache.org/";><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a> + </div> + <div class="searchbox"> + <form action="http://www.google.com/search"; method="get"> + <input value="zookeeper.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google"> + <input name="Search" value="Search" type="submit"> + </form> + </div> + <ul id="tabs"> + <li> + <a class="unselected" href="http://zookeeper.apache.org/";>Project</a> + </li> + <li> + <a class="unselected" href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/";>Wiki</a> + </li> + <li class="current"> + <a class="selected" href="index.html">ZooKeeper 3.8 Documentation</a> + </li> + </ul> + </div> +</div> +<div id="main"> + <div id="publishedStrip"> + <div id="level2tabs"></div> + <script type="text/javascript"><!-- +document.write("Last Published: " + document.lastModified); +// --></script> + </div> + <div class="breadtrail"> + + </div> + <div id="menu"> + <div onclick="SwitchMenu('menu_1', 'skin/')" id="menu_1Title" class="menutitle">Overview</div> + <div id="menu_1" class="menuitemgroup"> + <div class="menuitem"> + <a href="index.html">Welcome</a> + </div> + <div class="menuitem"> + <a href="zookeeperOver.html">Overview</a> + </div> + <div class="menuitem"> + <a href="zookeeperStarted.html">Getting Started</a> + </div> + <div class="menuitem"> + <a href="releasenotes.html">Release Notes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_2', 'skin/')" id="menu_2Title" class="menutitle">Developer</div> + <div id="menu_2" class="menuitemgroup"> + <div class="menuitem"> + <a href="apidocs/zookeeper-server/index.html">API Docs</a> + </div> + <div class="menuitem"> + <a href="zookeeperProgrammers.html">Programmer's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperUseCases.html">Use Cases</a> + </div> + <div class="menuitem"> + <a href="javaExample.html">Java Example</a> + </div> + <div class="menuitem"> + <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a> + </div> + <div class="menuitem"> + <a href="recipes.html">Recipes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_3', 'skin/')" id="menu_3Title" class="menutitle">Admin & Ops</div> + <div id="menu_3" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperAdmin.html">Administrator's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperQuotas.html">Quota Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperSnapshotAndRestore.html">Snapshot and Restore Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperJMX.html">JMX</a> + </div> + <div class="menuitem"> + <a href="zookeeperHierarchicalQuorums.html">Hierarchical Quorums</a> + </div> + <div class="menuitem"> + <a href="zookeeperOracleQuorums.html">Oracle Quorum</a> + </div> + <div class="menuitem"> + <a href="zookeeperObservers.html">Observers Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperReconfig.html">Dynamic Reconfiguration</a> + </div> + <div class="menuitem"> + <a href="zookeeperCLI.html">ZooKeeper CLI</a> + </div> + <div class="menuitem"> + <a href="zookeeperTools.html">ZooKeeper Tools</a> + </div> + <div class="menuitem"> + <a href="zookeeperMonitor.html">ZooKeeper Monitor</a> + </div> + <div class="menuitem"> + <a href="zookeeperAuditLogs.html">Audit Logs</a> + </div> + </div> + <div onclick="SwitchMenu('menu_4', 'skin/')" id="menu_4Title" class="menutitle">Contributor</div> + <div id="menu_4" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperInternals.html">ZooKeeper Internals</a> + </div> + </div> + <div onclick="SwitchMenu('menu_5', 'skin/')" id="menu_5Title" class="menutitle">Miscellaneous</div> + <div id="menu_5" class="menuitemgroup"> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER";>Wiki</a> + </div> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ";>FAQ</a> + </div> + <div class="menuitem"> + <a href="http://zookeeper.apache.org/mailing_lists.html";>Mailing Lists</a> + </div> + </div> + </div> + <div id="content"> +<!-- +Copyright 2002-2021 The Apache Software Foundation + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +//--> +<h1>ZooKeeper Monitor Guide</h1> +<ul> +<li> +<p><a href="#Metrics-System">New Metrics System</a></p> +<ul> +<li><a href="#Metrics">Metrics</a></li> +<li><a href="#Prometheus">Prometheus</a></li> +<li><a href="#Alerting">Alerting with Prometheus</a></li> +<li><a href="#Grafana">Grafana</a></li> +<li><a href="#influxdb">InfluxDB</a></li> +</ul> +</li> +<li> +<p><a href="#JMX">JMX</a></p> +</li> +<li> +<p><a href="#four-letter-words">Four letter words</a></p> +</li> +</ul> +<p><a name="Metrics-System"></a></p> +<h2>New Metrics System</h2> +<p>The feature:<code>New Metrics System</code> has been available since 3.6.0 which provides the abundant metrics to help users monitor the ZooKeeper on the topic: znode, network, disk, quorum, leader election, client, security, failures, watch/session, requestProcessor, and so forth.</p> +<p><a name="Metrics"></a></p> +<h3>Metrics</h3> +<p>All the metrics are included in the <code>ServerMetrics.java</code>.</p> +<p><a name="Prometheus"></a></p> +<h3>Prometheus</h3> +<ul> +<li>Running a <a href="https://prometheus.io/";>Prometheus</a> monitoring service is the easiest way to ingest and record ZooKeeper's metrics.</li> +<li>Pre-requisites:</li> +<li>enable the <code>Prometheus MetricsProvider</code> by setting <code>metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider</code> in the zoo.cfg.</li> +<li>the Port is also configurable by setting <code>metricsProvider.httpPort</code>ï¼the default value:7000ï¼</li> +<li>Install Prometheus: Go to the official website download <a href="https://prometheus.io/download/";>page</a>, download the latest release.</li> +<li> +<p>Set Prometheus's scraper to target the ZooKeeper cluster endpoints:</p> +<pre><code class="language-bash">cat > /tmp/test-zk.yaml <<EOF +global: + scrape_interval: 10s +scrape_configs: + - job_name: test-zk + static_configs: + - targets: ['192.168.10.32:7000','192.168.10.33:7000','192.168.10.34:7000'] +EOF +cat /tmp/test-zk.yaml +</code></pre> +</li> +<li> +<p>Set up the Prometheus handler:</p> +<pre><code class="language-bash">nohup /tmp/prometheus \ + --config.file /tmp/test-zk.yaml \ + --web.listen-address ":9090" \ + --storage.tsdb.path "/tmp/test-zk.data" >> /tmp/test-zk.log 2>&1 & +</code></pre> +</li> +<li> +<p>Now Prometheus will scrape zk metrics every 10 seconds.</p> +</li> +</ul> +<p><a name="Alerting"></a></p> +<h3>Alerting with Prometheus</h3> +<ul> +<li> +<p>We recommend that you read <a href="https://prometheus.io/docs/practices/alerting/";>Prometheus Official Alerting Page</a> to explore some principles of alerting</p> +</li> +<li> +<p>We recommend that you use <a href="https://www.prometheus.io/docs/alerting/latest/alertmanager/";>Prometheus Alertmanager</a> which can help users to receive alerting email or instant message(by webhook) in a more convenient way</p> +</li> +<li> +<p>We provide an alerting example where these metrics should be taken a special attention. Note: this is for your reference only, and you need to adjust them according to your actual situation and resource environment</p> +<pre><code>use ./promtool check rules rules/zk.yml to check the correctness of the config file +cat rules/zk.yml + +groups: +- name: zk-alert-example + rules: + - alert: ZooKeeper server is down + expr: up == 0 + for: 1m + labels: + severity: critical + annotations: + summary: "Instance {{ $labels.instance }} ZooKeeper server is down" + description: "{{ $labels.instance }} of job {{$labels.job}} ZooKeeper server is down: [{{ $value }}]." + + - alert: create too many znodes + expr: znode_count > 1000000 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} create too many znodes" + description: "{{ $labels.instance }} of job {{$labels.job}} create too many znodes: [{{ $value }}]." + + - alert: create too many connections + expr: num_alive_connections > 50 # suppose we use the default maxClientCnxns: 60 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} create too many connections" + description: "{{ $labels.instance }} of job {{$labels.job}} create too many connections: [{{ $value }}]." + + - alert: znode total occupied memory is too big + expr: approximate_data_size /1024 /1024 > 1 * 1024 # more than 1024 MB(1 GB) + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} znode total occupied memory is too big" + description: "{{ $labels.instance }} of job {{$labels.job}} znode total occupied memory is too big: [{{ $value }}] MB." + + - alert: set too many watch + expr: watch_count > 10000 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} set too many watch" + description: "{{ $labels.instance }} of job {{$labels.job}} set too many watch: [{{ $value }}]." + + - alert: a leader election happens + expr: increase(election_time_count[5m]) > 0 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} a leader election happens" + description: "{{ $labels.instance }} of job {{$labels.job}} a leader election happens: [{{ $value }}]." + + - alert: open too many files + expr: open_file_descriptor_count > 300 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} open too many files" + description: "{{ $labels.instance }} of job {{$labels.job}} open too many files: [{{ $value }}]." + + - alert: fsync time is too long + expr: rate(fsynctime_sum[1m]) > 100 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} fsync time is too long" + description: "{{ $labels.instance }} of job {{$labels.job}} fsync time is too long: [{{ $value }}]." + + - alert: take snapshot time is too long + expr: rate(snapshottime_sum[5m]) > 100 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} take snapshot time is too long" + description: "{{ $labels.instance }} of job {{$labels.job}} take snapshot time is too long: [{{ $value }}]." + + - alert: avg latency is too high + expr: avg_latency > 100 + for: 1m + labels: + severity: warning + annotations: + summary: "Instance {{ $labels.instance }} avg latency is too high" + description: "{{ $labels.instance }} of job {{$labels.job}} avg latency is too high: [{{ $value }}]." + + - alert: JvmMemoryFillingUp + expr: jvm_memory_bytes_used / jvm_memory_bytes_max{area="heap"} > 0.8 + for: 5m + labels: + severity: warning + annotations: + summary: "JVM memory filling up (instance {{ $labels.instance }})" + description: "JVM memory is filling up (> 80%)\n labels: {{ $labels }} value = {{ $value }}\n" +</code></pre> +</li> +</ul> +<p><a name="Grafana"></a></p> +<h3>Grafana</h3> +<ul> +<li>Grafana has built-in Prometheus support; just add a Prometheus data source: +<pre><code class="language-bash">Name: test-zk +Type: Prometheus +Url: http://localhost:9090 +Access: proxy +</code></pre> +</li> +<li>Then download and import the default ZooKeeper dashboard <a href="https://grafana.com/grafana/dashboards/10465";>template</a> and customize.</li> +<li>Users can ask for Grafana dashboard account if having any good improvements by writing a email to <strong>d...@zookeeper.apache.org</strong>.</li> +</ul> +<p><a name="influxdb"></a></p> +<h3>InfluxDB</h3> +<p>InfluxDB is an open source time series data that is often used to store metrics from Zookeeper. You can <a href="https://portal.influxdata.com/downloads/";>download</a> the open source version or create a <a href="https://cloud2.influxdata.com/signup";>free</a> account on InfluxDB Cloud. In either case, configure the <a href="https://www.influxdata.com/integration/apache-zookeeper/";>Apache Zookeeper Telegraf plugin</a> to start collecting and storing metrics from your Zookeeper clusters into your InfluxDB instance. There is also an <a href="https://www.influxdata.com/influxdb-templates/zookeeper-monitor/";>Apache Zookeeper InfluxDB template</a> that includes the Telegraf configurations and a dashboard to get you set up right away.</p> +<p><a name="JMX"></a></p> +<h2>JMX</h2> +<p>More details can be found in <a href="http://zookeeper.apache.org/doc/current/zookeeperJMX.html";>here</a></p> +<p><a name="four-letter-words"></a></p> +<h2>Four letter words</h2> +<p>More details can be found in <a href="http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkCommands";>here</a></p> +</div> +<div class="clearboth"> </div> +</div> +<div id="footer"> + <div class="lastmodified"> + <script type="text/javascript"> + <!-- + document.write("Last Published: " + document.lastModified); + // --> + </script> + </div> + <div class="copyright"> + Copyright © <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> + </div> + <div id="logos"></div> +</div> +</body> +</html> \ No newline at end of file Added: dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperObservers.html ============================================================================== --- dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperObservers.html (added) +++ dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperObservers.html Mon Jul 17 12:57:54 2023 @@ -0,0 +1,219 @@ + +<!DOCTYPE html> +<html> +<head> + <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>ZooKeeper: Because Coordinating Distributed Systems is a Zoo</title> + <link type="text/css" href="skin/basic.css" rel="stylesheet"> + <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet"> + <link media="print" type="text/css" href="skin/print.css" rel="stylesheet"> + <link type="text/css" href="skin/profile.css" rel="stylesheet"> + <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script> + <script src="skin/getMenu.js" language="javascript" type="text/javascript"></script> + <script src="skin/init.js" language="javascript" type="text/javascript"></script> + <link rel="shortcut icon" href="images/favicon.ico"> +</head> +<body onload="init();"> +<div id="top"> + <div class="breadtrail"> + <a href="http://www.apache.org/";>Apache</a> > <a href="http://zookeeper.apache.org/";>ZooKeeper</a> + </div> + <div class="header"> + <div class="projectlogo"> + <a href="http://zookeeper.apache.org/";><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a> + </div> + <div class="searchbox"> + <form action="http://www.google.com/search"; method="get"> + <input value="zookeeper.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google"> + <input name="Search" value="Search" type="submit"> + </form> + </div> + <ul id="tabs"> + <li> + <a class="unselected" href="http://zookeeper.apache.org/";>Project</a> + </li> + <li> + <a class="unselected" href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/";>Wiki</a> + </li> + <li class="current"> + <a class="selected" href="index.html">ZooKeeper 3.8 Documentation</a> + </li> + </ul> + </div> +</div> +<div id="main"> + <div id="publishedStrip"> + <div id="level2tabs"></div> + <script type="text/javascript"><!-- +document.write("Last Published: " + document.lastModified); +// --></script> + </div> + <div class="breadtrail"> + + </div> + <div id="menu"> + <div onclick="SwitchMenu('menu_1', 'skin/')" id="menu_1Title" class="menutitle">Overview</div> + <div id="menu_1" class="menuitemgroup"> + <div class="menuitem"> + <a href="index.html">Welcome</a> + </div> + <div class="menuitem"> + <a href="zookeeperOver.html">Overview</a> + </div> + <div class="menuitem"> + <a href="zookeeperStarted.html">Getting Started</a> + </div> + <div class="menuitem"> + <a href="releasenotes.html">Release Notes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_2', 'skin/')" id="menu_2Title" class="menutitle">Developer</div> + <div id="menu_2" class="menuitemgroup"> + <div class="menuitem"> + <a href="apidocs/zookeeper-server/index.html">API Docs</a> + </div> + <div class="menuitem"> + <a href="zookeeperProgrammers.html">Programmer's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperUseCases.html">Use Cases</a> + </div> + <div class="menuitem"> + <a href="javaExample.html">Java Example</a> + </div> + <div class="menuitem"> + <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a> + </div> + <div class="menuitem"> + <a href="recipes.html">Recipes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_3', 'skin/')" id="menu_3Title" class="menutitle">Admin & Ops</div> + <div id="menu_3" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperAdmin.html">Administrator's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperQuotas.html">Quota Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperSnapshotAndRestore.html">Snapshot and Restore Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperJMX.html">JMX</a> + </div> + <div class="menuitem"> + <a href="zookeeperHierarchicalQuorums.html">Hierarchical Quorums</a> + </div> + <div class="menuitem"> + <a href="zookeeperOracleQuorums.html">Oracle Quorum</a> + </div> + <div class="menuitem"> + <a href="zookeeperObservers.html">Observers Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperReconfig.html">Dynamic Reconfiguration</a> + </div> + <div class="menuitem"> + <a href="zookeeperCLI.html">ZooKeeper CLI</a> + </div> + <div class="menuitem"> + <a href="zookeeperTools.html">ZooKeeper Tools</a> + </div> + <div class="menuitem"> + <a href="zookeeperMonitor.html">ZooKeeper Monitor</a> + </div> + <div class="menuitem"> + <a href="zookeeperAuditLogs.html">Audit Logs</a> + </div> + </div> + <div onclick="SwitchMenu('menu_4', 'skin/')" id="menu_4Title" class="menutitle">Contributor</div> + <div id="menu_4" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperInternals.html">ZooKeeper Internals</a> + </div> + </div> + <div onclick="SwitchMenu('menu_5', 'skin/')" id="menu_5Title" class="menutitle">Miscellaneous</div> + <div id="menu_5" class="menuitemgroup"> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER";>Wiki</a> + </div> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ";>FAQ</a> + </div> + <div class="menuitem"> + <a href="http://zookeeper.apache.org/mailing_lists.html";>Mailing Lists</a> + </div> + </div> + </div> + <div id="content"> +<!-- +Copyright 2002-2004 The Apache Software Foundation + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +//--> +<h1>ZooKeeper Observers</h1> +<ul> +<li><a href="#ch_Introduction">Observers: Scaling ZooKeeper Without Hurting Write Performance</a></li> +<li><a href="#sc_UsingObservers">How to use Observers</a></li> +<li><a href="#ch_UseCases">Example use cases</a></li> +</ul> +<p><a name="ch_Introduction"></a></p> +<h2>Observers: Scaling ZooKeeper Without Hurting Write Performance</h2> +<p>Although ZooKeeper performs very well by having clients connect directly to voting members of the ensemble, this architecture makes it hard to scale out to huge numbers of clients. The problem is that as we add more voting members, the write performance drops. This is due to the fact that a write operation requires the agreement of (in general) at least half the nodes in an ensemble and therefore the cost of a vote can increase significantly as more voters are added.</p> +<p>We have introduced a new type of ZooKeeper node called an <em>Observer</em> which helps address this problem and further improves ZooKeeper's scalability. Observers are non-voting members of an ensemble which only hear the results of votes, not the agreement protocol that leads up to them. Other than this simple distinction, Observers function exactly the same as Followers - clients may connect to them and send read and write requests to them. Observers forward these requests to the Leader like Followers do, but they then simply wait to hear the result of the vote. Because of this, we can increase the number of Observers as much as we like without harming the performance of votes.</p> +<p>Observers have other advantages. Because they do not vote, they are not a critical part of the ZooKeeper ensemble. Therefore they can fail, or be disconnected from the cluster, without harming the availability of the ZooKeeper service. The benefit to the user is that Observers may connect over less reliable network links than Followers. In fact, Observers may be used to talk to a ZooKeeper server from another data center. Clients of the Observer will see fast reads, as all reads are served locally, and writes result in minimal network traffic as the number of messages required in the absence of the vote protocol is smaller.</p> +<p><a name="sc_UsingObservers"></a></p> +<h2>How to use Observers</h2> +<p>Setting up a ZooKeeper ensemble that uses Observers is very simple, and requires just two changes to your config files. Firstly, in the config file of every node that is to be an Observer, you must place this line:</p> +<pre><code>peerType=observer +</code></pre> +<p>This line tells ZooKeeper that the server is to be an Observer. Secondly, in every server config file, you must add :observer to the server definition line of each Observer. For example:</p> +<pre><code>server.1:localhost:2181:3181:observer +</code></pre> +<p>This tells every other server that server.1 is an Observer, and that they should not expect it to vote. This is all the configuration you need to do to add an Observer to your ZooKeeper cluster. Now you can connect to it as though it were an ordinary Follower. Try it out, by running:</p> +<pre><code>$ bin/zkCli.sh -server localhost:2181 +</code></pre> +<p>where localhost:2181 is the hostname and port number of the Observer as specified in every config file. You should see a command line prompt through which you can issue commands like <em>ls</em> to query the ZooKeeper service.</p> +<p><a name="ch_ObserverMasters"></a></p> +<h2>How to use Observer Masters</h2> +<p>Observers function simple as non-voting members of the ensemble, sharing the Learner interface with Followers and holding only a slightly different internal pipeline. Both maintain connections along the quorum port with the Leader by which they learn of all new proposals on the ensemble.</p> +<p>By default, Observers connect to the Leader of the quorum along its quorum port and this is how they learn of all new proposals on the ensemble. There are benefits to allowing Observers to connect to the Followers instead as a means of plugging into the commit stream in place of connecting to the Leader. It shifts the burden of supporting Observers off the Leader and allow it to focus on coordinating the commit of writes. This means better performance when the Leader is under high load, particularly high network load such as can happen after a leader election when many Learners need to sync. It reduces the total network connections maintained on the Leader when there are a high number of observers. Activating Followers to support Observers allow the overall number of Observers to scale into the hundreds. On the other end, Observer availability is improved since it will take shorter time for a high number of Observers to finish syncing and start serving client traffic.</p> +<p>This feature can be activated by letting all members of the ensemble know which port will be used by the Followers to listen for Observer connections. The following entry, when added to the server config file, will instruct Observers to connect to peers (Leaders and Followers) on port 2191 and instruct Followers to create an ObserverMaster thread to listen and serve on that port.</p> +<pre><code>observerMasterPort=2191 +</code></pre> +<p><a name="ch_UseCases"></a></p> +<h2>Example use cases</h2> +<p>Two example use cases for Observers are listed below. In fact, wherever you wish to scale the number of clients of your ZooKeeper ensemble, or where you wish to insulate the critical part of an ensemble from the load of dealing with client requests, Observers are a good architectural choice.</p> +<ul> +<li>As a datacenter bridge: Forming a ZK ensemble between two datacenters is a problematic endeavour as the high variance in latency between the datacenters could lead to false positive failure detection and partitioning. However if the ensemble runs entirely in one datacenter, and the second datacenter runs only Observers, partitions aren't problematic as the ensemble remains connected. Clients of the Observers may still see and issue proposals.</li> +<li>As a link to a message bus: Some companies have expressed an interest in using ZK as a component of a persistent reliable message bus. Observers would give a natural integration point for this work: a plug-in mechanism could be used to attach the stream of proposals an Observer sees to a publish-subscribe system, again without loading the core ensemble.</li> +</ul> +</div> +<div class="clearboth"> </div> +</div> +<div id="footer"> + <div class="lastmodified"> + <script type="text/javascript"> + <!-- + document.write("Last Published: " + document.lastModified); + // --> + </script> + </div> + <div class="copyright"> + Copyright © <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> + </div> + <div id="logos"></div> +</div> +</body> +</html> \ No newline at end of file Added: dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperOracleQuorums.html ============================================================================== --- dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperOracleQuorums.html (added) +++ dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperOracleQuorums.html Mon Jul 17 12:57:54 2023 @@ -0,0 +1,306 @@ + +<!DOCTYPE html> +<html> +<head> + <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>ZooKeeper: Because Coordinating Distributed Systems is a Zoo</title> + <link type="text/css" href="skin/basic.css" rel="stylesheet"> + <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet"> + <link media="print" type="text/css" href="skin/print.css" rel="stylesheet"> + <link type="text/css" href="skin/profile.css" rel="stylesheet"> + <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script> + <script src="skin/getMenu.js" language="javascript" type="text/javascript"></script> + <script src="skin/init.js" language="javascript" type="text/javascript"></script> + <link rel="shortcut icon" href="images/favicon.ico"> +</head> +<body onload="init();"> +<div id="top"> + <div class="breadtrail"> + <a href="http://www.apache.org/";>Apache</a> > <a href="http://zookeeper.apache.org/";>ZooKeeper</a> + </div> + <div class="header"> + <div class="projectlogo"> + <a href="http://zookeeper.apache.org/";><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a> + </div> + <div class="searchbox"> + <form action="http://www.google.com/search"; method="get"> + <input value="zookeeper.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google"> + <input name="Search" value="Search" type="submit"> + </form> + </div> + <ul id="tabs"> + <li> + <a class="unselected" href="http://zookeeper.apache.org/";>Project</a> + </li> + <li> + <a class="unselected" href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/";>Wiki</a> + </li> + <li class="current"> + <a class="selected" href="index.html">ZooKeeper 3.8 Documentation</a> + </li> + </ul> + </div> +</div> +<div id="main"> + <div id="publishedStrip"> + <div id="level2tabs"></div> + <script type="text/javascript"><!-- +document.write("Last Published: " + document.lastModified); +// --></script> + </div> + <div class="breadtrail"> + + </div> + <div id="menu"> + <div onclick="SwitchMenu('menu_1', 'skin/')" id="menu_1Title" class="menutitle">Overview</div> + <div id="menu_1" class="menuitemgroup"> + <div class="menuitem"> + <a href="index.html">Welcome</a> + </div> + <div class="menuitem"> + <a href="zookeeperOver.html">Overview</a> + </div> + <div class="menuitem"> + <a href="zookeeperStarted.html">Getting Started</a> + </div> + <div class="menuitem"> + <a href="releasenotes.html">Release Notes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_2', 'skin/')" id="menu_2Title" class="menutitle">Developer</div> + <div id="menu_2" class="menuitemgroup"> + <div class="menuitem"> + <a href="apidocs/zookeeper-server/index.html">API Docs</a> + </div> + <div class="menuitem"> + <a href="zookeeperProgrammers.html">Programmer's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperUseCases.html">Use Cases</a> + </div> + <div class="menuitem"> + <a href="javaExample.html">Java Example</a> + </div> + <div class="menuitem"> + <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a> + </div> + <div class="menuitem"> + <a href="recipes.html">Recipes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_3', 'skin/')" id="menu_3Title" class="menutitle">Admin & Ops</div> + <div id="menu_3" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperAdmin.html">Administrator's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperQuotas.html">Quota Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperSnapshotAndRestore.html">Snapshot and Restore Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperJMX.html">JMX</a> + </div> + <div class="menuitem"> + <a href="zookeeperHierarchicalQuorums.html">Hierarchical Quorums</a> + </div> + <div class="menuitem"> + <a href="zookeeperOracleQuorums.html">Oracle Quorum</a> + </div> + <div class="menuitem"> + <a href="zookeeperObservers.html">Observers Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperReconfig.html">Dynamic Reconfiguration</a> + </div> + <div class="menuitem"> + <a href="zookeeperCLI.html">ZooKeeper CLI</a> + </div> + <div class="menuitem"> + <a href="zookeeperTools.html">ZooKeeper Tools</a> + </div> + <div class="menuitem"> + <a href="zookeeperMonitor.html">ZooKeeper Monitor</a> + </div> + <div class="menuitem"> + <a href="zookeeperAuditLogs.html">Audit Logs</a> + </div> + </div> + <div onclick="SwitchMenu('menu_4', 'skin/')" id="menu_4Title" class="menutitle">Contributor</div> + <div id="menu_4" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperInternals.html">ZooKeeper Internals</a> + </div> + </div> + <div onclick="SwitchMenu('menu_5', 'skin/')" id="menu_5Title" class="menutitle">Miscellaneous</div> + <div id="menu_5" class="menuitemgroup"> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER";>Wiki</a> + </div> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ";>FAQ</a> + </div> + <div class="menuitem"> + <a href="http://zookeeper.apache.org/mailing_lists.html";>Mailing Lists</a> + </div> + </div> + </div> + <div id="content"> +<!-- +Copyright 2002-2004 The Apache Software Foundation + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +//--> +<h1>Introduction to Oracle Quorum</h1> +<p>The introduction to Oracle Quorum increases the availability of a cluster of 2 ZooKeeper instances with a failure detector as known as the Oracle. The Oracle is designed to grant the permission to the instance which is the only remaining instance in a 2-instance configuration when the other instance is identified as faulty by the fail detector, the Oracle.</p> +<h2>The implementation of the Oracle</h2> +<p>Every instance shall access to a file which contains either 0 or 1 to indicate whether this instance is authorized by the Oracle. However, this design can be changed since the fail detector algorithms vary from each other. Therefore, ones can override the method of <em>askOracle()</em> in <em>QuorumOracleMaj</em> to adapt the preferred way of deciphering the message from the Oracle.</p> +<h2>The deployment contexts</h2> +<p>The Oracle is designed to increase the availability of a cluster of 2 ZooKeeper instances; thus, the size of the voting member is <strong>2</strong>. In other words, the Oracle solves the consensus problem of a possibility of faulty instance in a two-instance ensemble.</p> +<p>In the case that the size of the voting members exceeds 2, the expected way to make the Oracle work correctly is to reconfigure the size of the cluster when a faulty machine is identified. For example, with a configuration of 5 instances, when a faulty machine breaks the connection with the Leader, it is expected to have a <em>reconfig</em> client request to the cluster, which makes the cluster to re-form as the configuration of 4 instances. Therefore, once the size of the voting member equals to 2, the configuration falls into the problem domain which the Oracle is designed to address.</p> +<h2>How to deploy the Oracle in <em>zoo.cfg</em></h2> +<p>Regardless of the size of the cluster, the <em>oraclePath</em> must be configured at the time of the initialization, which is like other static parameters. The below shows the correct way to specify and enable the Oracle.</p> +<pre><code>oraclePath=/to/some/file +</code></pre> +<h4>An example of zoo.cfg:</h4> +<pre><code>dataDir=/data +dataLogDir=/datalog +tickTime=2000 +initLimit=5 +syncLimit=2 +autopurge.snapRetainCount=3 +autopurge.purgeInterval=0 +maxClientCnxns=60 +standaloneEnabled=true +admin.enableServer=true +oraclePath=/chassis/mastership +server.1=0.0.0.0:2888:3888;2181 +server.2=hw1:2888:3888;2181 +</code></pre> +<p>The QuorumOracleMaj is designed to read the result of a failure detector, which is written on a text file, the oracle file.<br /> +The configuration in the zoo.cfg like the following:</p> +<pre><code>oraclePath=/to/some/file +</code></pre> +<p>Suppose you have the result of the failure detector written on /some/path/result.txt, and then the correct configuration is the following:</p> +<pre><code>oraclePath=/some/path/result.txt +</code></pre> +<p>So, what is the correct content of the provided file? An example file can be created with the following command from the terminal:</p> +<pre><code>$echo 1 > /some/path/result.txt +</code></pre> +<p>Any equivalent files are suitable for the current implementation of QuorumOracleMaj. The number of oracle files should be equal to the number of ZooKeeper instances configured to enable the Oracle. In other words, each ZooKeeper instance should have its oracle file, and the files shall not be shared; otherwise, the issues in the next section will arise.</p> +<h2>What differs after the deployment of the Oracle enabled</h2> +<p>The <em>QuorumPeerConfig</em> will create an instance of <em>QuorumOracleMaj</em> instead of the default QuorumVerifier, <em>QuorumMaj</em> when it reads the <em>zoo.cfg</em> contains <em>oraclePath</em>. QuorumOracleMaj inheritances from QuorumMaj, and differs from its superclass by overriding the method of <em>containsQuorum()</em>. QuorumOracleMaj is designed to execute its version of <em>containsQuorum</em> when the Leader loses all of its followers, and fails to maintain the quorum. In other cases, <em>QuorumOracleMaj</em> shall execute as <em>QuorumMaj</em>.</p> +<h2>What we should pay attention to the Oracle</h2> +<p>We consider an asynchronous distributed system which consists of <strong>2</strong> ZooKeeper instances and an Oracle.</p> +<h3>Liveness Issue:</h3> +<p>When we consider the oracle satisfies the following property introduced by [CT]:</p> +<pre><code>Strong Completeness: There is a time after which every process that crashes is permanently suspected by every correct processes +</code></pre> +<p>The liveness of the system is ensured by the Oracle. However, when the introduced oracle fails to maintain this property, the lost of the liveness is expected as the following example,</p> +<p>Suppose we have a Leader and a Follower, which are running in the broadcasting state, The system will lose its liveness when:</p> +<ol> +<li>The Leader fails, but the Oracle does not detect the faulty Leader, which means the Oracle will not authorize the Follower to become a new Leader. +<ol> +<li>When a Follower fails, but the Oracle does not detect the faulty follower, which means the Oracle will authorize the Leader to move system forward.</li> +</ol> +</li> +</ol> +<h3>Safety Issue:</h3> +<h4>Lost of Progress</h4> +<p>The progress can lost when multiple failures occurs in the system at different time as the following example,</p> +<p>Suppose we have a Leader(Ben) and a Follower(John) in the broadcasting state,</p> +<pre><code>At T1 with zxid(0x1_1): L-Ben fails, and the F-John takes over the system under the authorization from the Oracle. +At T2 with zxid(0x2_1): The F-John becomes a new Leader, L-John, and starts a new epoch. +At T3 with zxid(0x2_A): L-John fails +At T4 with zxid(0x2_A): Ben recovers up and starts its leader election. +At T5 with zxid(0x3_1): Ben becomes the new leader, L-Ben, under the authorization from the Oracle. +</code></pre> +<p>In this case, the system loses its progress after the L-Ben failed.</p> +<p>However, the lost of progress can be prevented by making the Oracle is capable of referring the latest zxid. When the Oracle could refer to the latest zxid,</p> +<pre><code>At T5 with zxid(0x2_A): Ben will not end his leader election because the Oracle would not authorize although John is down. +</code></pre> +<p>Nevertheless, we exchange the liveness for the safety.</p> +<h4>Split Brain Issue</h4> +<p>We consider the Oracle satisfies the following desired property introduced by [CT],</p> +<pre><code>Accuracy: There is a time after which some correct processes is never suspected by any processes +</code></pre> +<p>Nevertheless, the decisions which the Oracle gives out should be mutual exclusive.</p> +<p>In other words,</p> +<p>Suppose we have a Leader(Ben) and a Follower(John) in the broadcasting state,</p> +<ul> +<li>At any time, the Oracle will not authorize both Ben and John even though the failure detectors think each other is faulty. Or +<ul> +<li>At any time, for any two values in any two Oracle files respectively, the values are not both equal to 1.</li> +</ul> +</li> +</ul> +<p>The split brain is expected when the Oracle fails to maintain this property during the leader election phase of</p> +<ol> +<li>Start of the system +<ol> +<li>A failed instance recovers from failures.</li> +</ol> +</li> +</ol> +<h2>Examples of Concepts for Implementation of a Failure Detector</h2> +<p>One should consider that the failure detector's outcome is to authorize the querying ZooKeeper instance whether it has the right to move the system forward without waiting for the faulty instance, which is identified by the failure detector.</p> +<h3>An Implementation of Hardware</h3> +<p>Suppose two dedicated pieces of hardware, hw1 and hw2, can host ZooKeeper instances, zk1 and zk2, respectively, and form a cluster. A hardware device is attached to both of the hardware, and it is capable of determining whether the hardware is power on or not. So, when hw1 is not power on, the zk1 is undoubtedly faulty. Therefore, the hardware device updates the oracle file on hw2 to 1, which indicates that zk1 is faulty and authorizes zk2 to move the system forwards.</p> +<h3>An Implementation of Software</h3> +<p>Suppose two dedicated pieces of hardware, hw1 and hw2, can host ZooKeeper instances, zk1 and zk2, respectively, and form a cluster. One can have two more services, o1 and o2, on hw1 and hw2, respectively. The job of o1 and o2 are detecting the other hardware is alive or not. For example, o1 can constantly ping hw2 to determine if hw2 is power on or not. When o1 cannot ping hw2, o1 identifies that hw2 is faulty and then update the oracle file of zk1 to 1, which indicates that zk2 is faulty and authorizes zk1 to move the system forwards.</p> +<h3>Use USB devices as Oracle to Maintain Progress</h3> +<p>In macOS,10.15.7 (19H2), the external storage devices are mounted under <code>/Volumes</code>. Thus, we can insert a USB device which contains the required information as the oracle. When the device is connected, the oracle authorizes the leader to move system forward, which also means the other instance fails. There are <strong>SIX</strong> steps to reproduce this stimulation.</p> +<ul> +<li>Firstly, insert a USB device named <code>Oracle</code>, and then we can expect that <code>/Volumes/Oracle</code> is accessible.</li> +<li>Secondly, we create a file contains <code>1</code> under <code>/Volumes/Oracle</code> named <code>mastership</code>. Now we can access <code>/Volumes/Oracle/mastership</code>, and so does the zookeeper instances to see whether it has the right to move the system forward. The file can easily be generated by the following command: +<p>$echo 1 > mastership</p> +</li> +<li> +<p>Thirdly, you shall have a <code>zoo.cfg</code> like the example below:</p> +<p>dataDir=/data dataLogDir=/datalog tickTime=2000 initLimit=5 syncLimit=2 autopurge.snapRetainCount=3 autopurge.purgeInterval=0 maxClientCnxns=60 standaloneEnabled=true admin.enableServer=true oraclePath=/Volumes/Oracle/mastership server.1=0.0.0.0:2888:3888;2181 server.2=hw1:2888:3888;2181</p> +</li> +</ul> +<p><em>(NOTE) The split brain issues will not occur because there is only a SINGLE USB device in this stimulation.</em> <em>Additionally, <code>mastership</code> should not be shared by multiple instances.</em> <em>Thus, only one ZooKeeper instance is configured with Oracle.</em> <em>For more, please refer to Section Safety Issue.</em></p> +<ul> +<li>Fourthly, start the cluster, and it is expected it forms a quorum normally.</li> +<li>Fifthly, terminate the instance either without attaching to a USB device or <code>mastership</code> contains 0. There are two scenarios to expect:</li> +</ul> +<ol> +<li>A leader failure occurs, and the remained instance finishes the leader election on its own due to the oracle.</li> +<li>The quorum is still maintained due to the oracle.</li> +</ol> +<ul> +<li>Lastly, when the USB device is removed, <code>/Volumes/Oracle/mastership</code> becomes unavailable. Therefore, according to the current implementation, whenever the Leader queries the oracle, the oracle throws an exception and return <code>FALSE</code>. Repeat the fifth step, and then it is expected that either the system cannot recover from a leader failure ,or the leader loses the quorum. In either case, the service is interrupted.</li> +</ul> +<p>With these steps, we can show and practice how the oracle works with two-instance systems with ease.</p> +<h2>REFERENCE</h2> +<p>[CT] Tushar Deepak Chandra and Sam Toueg. 1991. Unreliable failure detectors for asynchronous systems (preliminary version). In <i>Proceedings of the tenth annual ACM symposium on Principles of distributed computing</i> (<i>PODC '91</i>). Association for Computing Machinery, New York, NY, USA, 325â340. DOI:https://doi.org/10.1145/112600.112627</p> +</div> +<div class="clearboth"> </div> +</div> +<div id="footer"> + <div class="lastmodified"> + <script type="text/javascript"> + <!-- + document.write("Last Published: " + document.lastModified); + // --> + </script> + </div> + <div class="copyright"> + Copyright © <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> + </div> + <div id="logos"></div> +</div> +</body> +</html> \ No newline at end of file Added: dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperOver.html ============================================================================== --- dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperOver.html (added) +++ dev/zookeeper/zookeeper-3.9.0-candidate-0/website/zookeeperOver.html Mon Jul 17 12:57:54 2023 @@ -0,0 +1,305 @@ + +<!DOCTYPE html> +<html> +<head> + <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> + <title>ZooKeeper: Because Coordinating Distributed Systems is a Zoo</title> + <link type="text/css" href="skin/basic.css" rel="stylesheet"> + <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet"> + <link media="print" type="text/css" href="skin/print.css" rel="stylesheet"> + <link type="text/css" href="skin/profile.css" rel="stylesheet"> + <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script> + <script src="skin/getMenu.js" language="javascript" type="text/javascript"></script> + <script src="skin/init.js" language="javascript" type="text/javascript"></script> + <link rel="shortcut icon" href="images/favicon.ico"> +</head> +<body onload="init();"> +<div id="top"> + <div class="breadtrail"> + <a href="http://www.apache.org/";>Apache</a> > <a href="http://zookeeper.apache.org/";>ZooKeeper</a> + </div> + <div class="header"> + <div class="projectlogo"> + <a href="http://zookeeper.apache.org/";><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a> + </div> + <div class="searchbox"> + <form action="http://www.google.com/search"; method="get"> + <input value="zookeeper.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google"> + <input name="Search" value="Search" type="submit"> + </form> + </div> + <ul id="tabs"> + <li> + <a class="unselected" href="http://zookeeper.apache.org/";>Project</a> + </li> + <li> + <a class="unselected" href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/";>Wiki</a> + </li> + <li class="current"> + <a class="selected" href="index.html">ZooKeeper 3.8 Documentation</a> + </li> + </ul> + </div> +</div> +<div id="main"> + <div id="publishedStrip"> + <div id="level2tabs"></div> + <script type="text/javascript"><!-- +document.write("Last Published: " + document.lastModified); +// --></script> + </div> + <div class="breadtrail"> + + </div> + <div id="menu"> + <div onclick="SwitchMenu('menu_1', 'skin/')" id="menu_1Title" class="menutitle">Overview</div> + <div id="menu_1" class="menuitemgroup"> + <div class="menuitem"> + <a href="index.html">Welcome</a> + </div> + <div class="menuitem"> + <a href="zookeeperOver.html">Overview</a> + </div> + <div class="menuitem"> + <a href="zookeeperStarted.html">Getting Started</a> + </div> + <div class="menuitem"> + <a href="releasenotes.html">Release Notes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_2', 'skin/')" id="menu_2Title" class="menutitle">Developer</div> + <div id="menu_2" class="menuitemgroup"> + <div class="menuitem"> + <a href="apidocs/zookeeper-server/index.html">API Docs</a> + </div> + <div class="menuitem"> + <a href="zookeeperProgrammers.html">Programmer's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperUseCases.html">Use Cases</a> + </div> + <div class="menuitem"> + <a href="javaExample.html">Java Example</a> + </div> + <div class="menuitem"> + <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a> + </div> + <div class="menuitem"> + <a href="recipes.html">Recipes</a> + </div> + </div> + <div onclick="SwitchMenu('menu_3', 'skin/')" id="menu_3Title" class="menutitle">Admin & Ops</div> + <div id="menu_3" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperAdmin.html">Administrator's Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperQuotas.html">Quota Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperSnapshotAndRestore.html">Snapshot and Restore Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperJMX.html">JMX</a> + </div> + <div class="menuitem"> + <a href="zookeeperHierarchicalQuorums.html">Hierarchical Quorums</a> + </div> + <div class="menuitem"> + <a href="zookeeperOracleQuorums.html">Oracle Quorum</a> + </div> + <div class="menuitem"> + <a href="zookeeperObservers.html">Observers Guide</a> + </div> + <div class="menuitem"> + <a href="zookeeperReconfig.html">Dynamic Reconfiguration</a> + </div> + <div class="menuitem"> + <a href="zookeeperCLI.html">ZooKeeper CLI</a> + </div> + <div class="menuitem"> + <a href="zookeeperTools.html">ZooKeeper Tools</a> + </div> + <div class="menuitem"> + <a href="zookeeperMonitor.html">ZooKeeper Monitor</a> + </div> + <div class="menuitem"> + <a href="zookeeperAuditLogs.html">Audit Logs</a> + </div> + </div> + <div onclick="SwitchMenu('menu_4', 'skin/')" id="menu_4Title" class="menutitle">Contributor</div> + <div id="menu_4" class="menuitemgroup"> + <div class="menuitem"> + <a href="zookeeperInternals.html">ZooKeeper Internals</a> + </div> + </div> + <div onclick="SwitchMenu('menu_5', 'skin/')" id="menu_5Title" class="menutitle">Miscellaneous</div> + <div id="menu_5" class="menuitemgroup"> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER";>Wiki</a> + </div> + <div class="menuitem"> + <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ";>FAQ</a> + </div> + <div class="menuitem"> + <a href="http://zookeeper.apache.org/mailing_lists.html";>Mailing Lists</a> + </div> + </div> + </div> + <div id="content"> +<!-- +Copyright 2002-2004 The Apache Software Foundation + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +//--> +<h1>ZooKeeper</h1> +<ul> +<li><a href="#ch_DesignOverview">ZooKeeper: A Distributed Coordination Service for Distributed Applications</a> +<ul> +<li><a href="#sc_designGoals">Design Goals</a></li> +<li><a href="#sc_dataModelNameSpace">Data model and the hierarchical namespace</a></li> +<li><a href="#Nodes+and+ephemeral+nodes">Nodes and ephemeral nodes</a></li> +<li><a href="#Conditional+updates+and+watches">Conditional updates and watches</a></li> +<li><a href="#Guarantees">Guarantees</a></li> +<li><a href="#Simple+API">Simple API</a></li> +<li><a href="#Implementation">Implementation</a></li> +<li><a href="#Uses">Uses</a></li> +<li><a href="#Performance">Performance</a></li> +<li><a href="#Reliability">Reliability</a></li> +<li><a href="#The+ZooKeeper+Project">The ZooKeeper Project</a></li> +</ul> +</li> +</ul> +<p><a name="ch_DesignOverview"></a></p> +<h2>ZooKeeper: A Distributed Coordination Service for Distributed Applications</h2> +<p>ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C.</p> +<p>Coordination services are notoriously hard to get right. They are especially prone to errors such as race conditions and deadlock. The motivation behind ZooKeeper is to relieve distributed applications the responsibility of implementing coordination services from scratch.</p> +<p><a name="sc_designGoals"></a></p> +<h3>Design Goals</h3> +<p><strong>ZooKeeper is simple.</strong> ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical namespace which is organized similarly to a standard file system. The namespace consists of data registers - called znodes, in ZooKeeper parlance - and these are similar to files and directories. Unlike a typical file system, which is designed for storage, ZooKeeper data is kept in-memory, which means ZooKeeper can achieve high throughput and low latency numbers.</p> +<p>The ZooKeeper implementation puts a premium on high performance, highly available, strictly ordered access. The performance aspects of ZooKeeper means it can be used in large, distributed systems. The reliability aspects keep it from being a single point of failure. The strict ordering means that sophisticated synchronization primitives can be implemented at the client.</p> +<p><strong>ZooKeeper is replicated.</strong> Like the distributed processes it coordinates, ZooKeeper itself is intended to be replicated over a set of hosts called an ensemble.</p> +<p><img src="images/zkservice.jpg" alt="ZooKeeper Service" /></p> +<p>The servers that make up the ZooKeeper service must all know about each other. They maintain an in-memory image of state, along with a transaction logs and snapshots in a persistent store. As long as a majority of the servers are available, the ZooKeeper service will be available.</p> +<p>Clients connect to a single ZooKeeper server. The client maintains a TCP connection through which it sends requests, gets responses, gets watch events, and sends heart beats. If the TCP connection to the server breaks, the client will connect to a different server.</p> +<p><strong>ZooKeeper is ordered.</strong> ZooKeeper stamps each update with a number that reflects the order of all ZooKeeper transactions. Subsequent operations can use the order to implement higher-level abstractions, such as synchronization primitives.</p> +<p><strong>ZooKeeper is fast.</strong> It is especially fast in "read-dominant" workloads. ZooKeeper applications run on thousands of machines, and it performs best where reads are more common than writes, at ratios of around 10:1.</p> +<p><a name="sc_dataModelNameSpace"></a></p> +<h3>Data model and the hierarchical namespace</h3> +<p>The namespace provided by ZooKeeper is much like that of a standard file system. A name is a sequence of path elements separated by a slash (/). Every node in ZooKeeper's namespace is identified by a path.</p> +<h4>ZooKeeper's Hierarchical Namespace</h4> +<p><img src="images/zknamespace.jpg" alt="ZooKeeper's Hierarchical Namespace" /></p> +<p><a name="Nodes+and+ephemeral+nodes"></a></p> +<h3>Nodes and ephemeral nodes</h3> +<p>Unlike standard file systems, each node in a ZooKeeper namespace can have data associated with it as well as children. It is like having a file-system that allows a file to also be a directory. (ZooKeeper was designed to store coordination data: status information, configuration, location information, etc., so the data stored at each node is usually small, in the byte to kilobyte range.) We use the term <em>znode</em> to make it clear that we are talking about ZooKeeper data nodes.</p> +<p>Znodes maintain a stat structure that includes version numbers for data changes, ACL changes, and timestamps, to allow cache validations and coordinated updates. Each time a znode's data changes, the version number increases. For instance, whenever a client retrieves data it also receives the version of the data.</p> +<p>The data stored at each znode in a namespace is read and written atomically. Reads get all the data bytes associated with a znode and a write replaces all the data. Each node has an Access Control List (ACL) that restricts who can do what.</p> +<p>ZooKeeper also has the notion of ephemeral nodes. These znodes exists as long as the session that created the znode is active. When the session ends the znode is deleted.</p> +<p><a name="Conditional+updates+and+watches"></a></p> +<h3>Conditional updates and watches</h3> +<p>ZooKeeper supports the concept of <em>watches</em>. Clients can set a watch on a znode. A watch will be triggered and removed when the znode changes. When a watch is triggered, the client receives a packet saying that the znode has changed. If the connection between the client and one of the ZooKeeper servers is broken, the client will receive a local notification.</p> +<p><strong>New in 3.6.0:</strong> Clients can also set permanent, recursive watches on a znode that are not removed when triggered and that trigger for changes on the registered znode as well as any children znodes recursively.</p> +<p><a name="Guarantees"></a></p> +<h3>Guarantees</h3> +<p>ZooKeeper is very fast and very simple. Since its goal, though, is to be a basis for the construction of more complicated services, such as synchronization, it provides a set of guarantees. These are:</p> +<ul> +<li>Sequential Consistency - Updates from a client will be applied in the order that they were sent.</li> +<li>Atomicity - Updates either succeed or fail. No partial results.</li> +<li>Single System Image - A client will see the same view of the service regardless of the server that it connects to. i.e., a client will never see an older view of the system even if the client fails over to a different server with the same session.</li> +<li>Reliability - Once an update has been applied, it will persist from that time forward until a client overwrites the update.</li> +<li>Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.</li> +</ul> +<p><a name="Simple+API"></a></p> +<h3>Simple API</h3> +<p>One of the design goals of ZooKeeper is providing a very simple programming interface. As a result, it supports only these operations:</p> +<ul> +<li> +<p><em>create</em> : creates a node at a location in the tree</p> +</li> +<li> +<p><em>delete</em> : deletes a node</p> +</li> +<li> +<p><em>exists</em> : tests if a node exists at a location</p> +</li> +<li> +<p><em>get data</em> : reads the data from a node</p> +</li> +<li> +<p><em>set data</em> : writes data to a node</p> +</li> +<li> +<p><em>get children</em> : retrieves a list of children of a node</p> +</li> +<li> +<p><em>sync</em> : waits for data to be propagated</p> +</li> +</ul> +<p><a name="Implementation"></a></p> +<h3>Implementation</h3> +<p><a href="#zkComponents">ZooKeeper Components</a> shows the high-level components of the ZooKeeper service. With the exception of the request processor, each of the servers that make up the ZooKeeper service replicates its own copy of each of the components.</p> +<p><a name="zkComponents"></a></p> +<p><img src="images/zkcomponents.jpg" alt="ZooKeeper Components" /></p> +<p>The replicated database is an in-memory database containing the entire data tree. Updates are logged to disk for recoverability, and writes are serialized to disk before they are applied to the in-memory database.</p> +<p>Every ZooKeeper server services clients. Clients connect to exactly one server to submit requests. Read requests are serviced from the local replica of each server database. Requests that change the state of the service, write requests, are processed by an agreement protocol.</p> +<p>As part of the agreement protocol all write requests from clients are forwarded to a single server, called the <em>leader</em>. The rest of the ZooKeeper servers, called <em>followers</em>, receive message proposals from the leader and agree upon message delivery. The messaging layer takes care of replacing leaders on failures and syncing followers with leaders.</p> +<p>ZooKeeper uses a custom atomic messaging protocol. Since the messaging layer is atomic, ZooKeeper can guarantee that the local replicas never diverge. When the leader receives a write request, it calculates what the state of the system is when the write is to be applied and transforms this into a transaction that captures this new state.</p> +<p><a name="Uses"></a></p> +<h3>Uses</h3> +<p>The programming interface to ZooKeeper is deliberately simple. With it, however, you can implement higher order operations, such as synchronizations primitives, group membership, ownership, etc.</p> +<p><a name="Performance"></a></p> +<h3>Performance</h3> +<p>ZooKeeper is designed to be highly performance. But is it? The results of the ZooKeeper's development team at Yahoo! Research indicate that it is. (See <a href="#zkPerfRW">ZooKeeper Throughput as the Read-Write Ratio Varies</a>.) It is especially high performance in applications where reads outnumber writes, since writes involve synchronizing the state of all servers. (Reads outnumbering writes is typically the case for a coordination service.)</p> +<p><a name="zkPerfRW"></a></p> +<p><img src="images/zkperfRW-3.2.jpg" alt="ZooKeeper Throughput as the Read-Write Ratio Varies" /></p> +<p>The <a href="#zkPerfRW">ZooKeeper Throughput as the Read-Write Ratio Varies</a> is a throughput graph of ZooKeeper release 3.2 running on servers with dual 2Ghz Xeon and two SATA 15K RPM drives. One drive was used as a dedicated ZooKeeper log device. The snapshots were written to the OS drive. Write requests were 1K writes and the reads were 1K reads. "Servers" indicate the size of the ZooKeeper ensemble, the number of servers that make up the service. Approximately 30 other servers were used to simulate the clients. The ZooKeeper ensemble was configured such that leaders do not allow connections from clients.</p> +<h6>Note</h6> +<blockquote> +<p>In version 3.2 r/w performance improved by ~2x compared to the <a href="http://zookeeper.apache.org/docs/r3.1.1/zookeeperOver.html#Performance";>previous 3.1 release</a>.</p> +</blockquote> +<p>Benchmarks also indicate that it is reliable, too. <a href="#zkPerfReliability">Reliability in the Presence of Errors</a> shows how a deployment responds to various failures. The events marked in the figure are the following:</p> +<ol> +<li>Failure and recovery of a follower</li> +<li>Failure and recovery of a different follower</li> +<li>Failure of the leader</li> +<li>Failure and recovery of two followers</li> +<li>Failure of another leader</li> +</ol> +<p><a name="Reliability"></a></p> +<h3>Reliability</h3> +<p>To show the behavior of the system over time as failures are injected we ran a ZooKeeper service made up of 7 machines. We ran the same saturation benchmark as before, but this time we kept the write percentage at a constant 30%, which is a conservative ratio of our expected workloads.</p> +<p><a name="zkPerfReliability"></a></p> +<p><img src="images/zkperfreliability.jpg" alt="Reliability in the Presence of Errors" /></p> +<p>There are a few important observations from this graph. First, if followers fail and recover quickly, then ZooKeeper is able to sustain a high throughput despite the failure. But maybe more importantly, the leader election algorithm allows for the system to recover fast enough to prevent throughput from dropping substantially. In our observations, ZooKeeper takes less than 200ms to elect a new leader. Third, as followers recover, ZooKeeper is able to raise throughput again once they start processing requests.</p> +<p><a name="The+ZooKeeper+Project"></a></p> +<h3>The ZooKeeper Project</h3> +<p>ZooKeeper has been <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy";>successfully used</a> in many industrial applications. It is used at Yahoo! as the coordination and failure recovery service for Yahoo! Message Broker, which is a highly scalable publish-subscribe system managing thousands of topics for replication and data delivery. It is used by the Fetching Service for Yahoo! crawler, where it also manages failure recovery. A number of Yahoo! advertising systems also use ZooKeeper to implement reliable services.</p> +<p>All users and developers are encouraged to join the community and contribute their expertise. See the <a href="http://zookeeper.apache.org/";>Zookeeper Project on Apache</a> for more information.</p> +</div> +<div class="clearboth"> </div> +</div> +<div id="footer"> + <div class="lastmodified"> + <script type="text/javascript"> + <!-- + document.write("Last Published: " + document.lastModified); + // --> + </script> + </div> + <div class="copyright"> + Copyright © <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> + </div> + <div id="logos"></div> +</div> +</body> +</html> \ No newline at end of file
