Added: websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperLedgers2Logs.html ============================================================================== --- websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperLedgers2Logs.html (added) +++ websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperLedgers2Logs.html Sun May 15 21:40:24 2016 @@ -0,0 +1,172 @@ +<!DOCTYPE html> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE- 2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> + <head> + <title>Apache BookKeeper - From Ledgers to Logs</title> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <!-- Bootstrap --> + <link href="/css/bootstrap.min.css" rel="stylesheet"> + <link href="/css/bootstrap-responsive.min.css" rel="stylesheet"> + <link href="/css/styles.css" rel="stylesheet"> + </head> + <body> + <header class="navbar navbar-inverse navbar-static-top" role="banner"> + <div class="container"> + <div class="navbar-header hidden-xs hidden-sm"> + <a class="navbar-brand navbar-logo" href="/"><img class="img-responsive" src="/img/bookkeeper_blk40.png" alt="Bookkeeper Logo" /></a> + </div> + <div class="navbar-header"> + <button class="navbar-toggle collapsed" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/">Apache BookKeeper</a> + </div> + <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> + <ul class="nav navbar-nav"> + <li><a href="/releases.html">Download</a></li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/docs/master">Latest (master)</a></li> + <li><ul> + <li><a href="/docs/master/apidocs">Java API docs</a></li> + <li><a href="/docs/master/bookkeeperTutorial.html">Tutorial</a></li> + <li><a href="/docs/master/bookkeeperConfig.html">Admin guide</a></li> + </ul><li> + <li><a href="/docs/r4.4.0">Release 4.4.0</a></li> + <li class="divider"></li> + <li>Older releases</li> + <li><a href="/docs/r4.3.2">Release 4.3.2</a></li> + <li><a href="/docs/r4.3.1">Release 4.3.1</a></li> + <li><a href="/docs/r4.3.0">Release 4.3.0</a></li> + <li><a href="/docs/r4.2.4">Release 4.2.4</a></li> + <li><a href="/docs/r4.2.3">Release 4.2.3</a></li> + <li><a href="/docs/r4.2.2">Release 4.2.2</a></li> + <li><a href="/docs/r4.2.1">Release 4.2.1</a></li> + <li><a href="/docs/r4.2.0">Release 4.2.0</a></li> + <li><a href="/docs/r4.1.0">Release 4.1.0</a></li> + <li><a href="/docs/r4.0.0">Release 4.0.0</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Get Involved<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/lists.html">Mailing Lists</a></li> + <li><a href="/irc.html">IRC</a></li> + <li><a href="/svn.html">Version Control</a></li> + <li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER";>Issue Tracker</a></li> + </ul> + </li> + + <li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index";>Wiki</a></li> + <!--<li><a href="#">Hedwig</a></li>//--> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project Info<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/credits.html">Who are we?</a></li> + <li><a href="/bylaws.html">Bylaws</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li class="divider"></li> + <li><a href="/privacy.html">Privacy Policy</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsership</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + <script> + (function() { + var cx = '017580107654524087317:iqnsyimpydg'; + var gcse = document.createElement('script'); + gcse.type = 'text/javascript'; + gcse.async = true; + gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + + '//www.google.com/cse/cse.js?cx=' + cx; + var s = document.getElementsByTagName('script')[0]; + s.parentNode.insertBefore(gcse, s); + })(); + </script> + + <div class="navbar-form navbar-right visible-lg" id="googlebox"> + <gcse:searchbox-only></gcse:searchbox-only> + </div> + </nav> + </div> + </header> + <div class="container"> + + <p>This documents describes the bookkeeper replication protocol, and the guarantees it gives. It assumes you have a general idea about leader election and log replication and how you can use these in your system. If not, have a look at the bookkeeper <a href="https://github.com/ivankelly/bookkeeper-tutorial";>tutorial</a> first.</p> + +<h1>Ledgers to Logs</h1> + +<p>Bookkeeper provides a primitive, ledgers, which can be used to build a replicated log for your system. All guarantees provided by bookkeeper are on ledgers. You can learn about the guarantees of ledgers <a href="./bookkeeperProtocol.html">here</a>. Guarantees on the whole log can be built using the ledger guarantees and any consistent datastore with a compare-and-swap(CAS) primitive. In this case, we describe a log using zookeeper as the datastore, but others could theoretically be used. </p> + +<p>A log in bookkeeper is built from a number of ledgers, with a fixed order. A ledger represents a single segment of the log. A ledger could be the whole period that one node was the leader, or there could be multiple ledgers for a single period of leadership. However, there can only ever been one leader that adds entries to a single ledger. Ledgers cannot be reopened for writing once they have been closed/recovered.</p> + +<p>It's important to note that bookkeeper doesn't provide leader election. You must use a system like Zookeeper for this.</p> + +<p>In many cases, leader election is really leader suggestion. Multiple nodes could think that they are leader at any one time. It is the job of the log to guarantee that only one can write changes to the system.</p> + +<h3>Opening a log</h3> + +<p>Once a node thinks it is leader for a particular log, it must take the following steps.</p> + +<ol> +<li>read the list of ledgers for the log</li> +<li>fence the last 2 ledgers<sup class="footnote"><a href="#fn1">1</a></sup> in the list</li> +<li>create a new ledger</li> +<li>add the new ledger to the ledger list</li> +<li>write the new ledger list back to the datastore using a <span class="caps">CAS </span>operation.</li> +</ol> + +<p>The fencing in step 2 and the compare-and-swap operation in step 5 prevents two nodes thinking they have leadership at any one time. Ledger fencing is described in <a href="./bookkeeperProtocol.html">Bookkeeper Protocol</a>. The compare-and-swap operation will fail if the list of ledgers has changed between reading it and writing back the new list. When the <span class="caps">CAS </span>operation fails, the leader must start at step 1 again. Even better, they should check that they are in fact still the leader with the system that is providing leader election. The protocol will work correctly without this step, though it will be able to make very little progress if two nodes think they are leader and are duelling for the log. </p> + +<p>The node must not serve any writes until step 5 completes successfully.</p> + +<h3>Rolling ledgers</h3> + +<p>The leader may wish to close the current ledger and open a new one every so often. Ledgers can only be deleted as a whole. If you don't roll the log, you won't be able to clean up old entries in the log without a leader change. By closing the current ledger and adding a new one, the leader allows the log to be truncated whenever that data is no longer needed. The steps for rolling the log is similar to those for creating a new ledger. </p> + +<ol> +<li>create a new ledger</li> +<li>add the new ledger to the ledger list</li> +<li>write the new ledger list to the datastore using <span class="caps">CAS</span></li> +<li>close the previous ledger</li> +</ol> + +<p>By deferring the closing of the previous ledger until step 4, we can continue writing to the log while we perform metadata update operations to add the new ledger. This is safe as long as you fence the last <em>2</em> ledgers when acquiring leadership.</p> + +<p class="footnote" id="fn1"><sup>1</sup> We fence 2 ledgers, as the write may be writing to the penultimate, while adding the last ledger to the ledger list.</p> + + </div> + <footer class="footer"> + <div class="container"> + <p class="text-muted">Copyright © 2014 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.<br/> + Apache BookKeeper, BookKeeper, Apache, Apache ZooKeeper, ZooKeeper, the Apache feather logo, and the Apache BookKeeper project logo are trademarks of The Apache Software Foundation.</p> + </div> + </footer> + + <script src="http://code.jquery.com/jquery.js";></script> + <script src="/js/bootstrap.min.js"></script> + </body> +</html>
Added: websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperMetadata.html ============================================================================== --- websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperMetadata.html (added) +++ websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperMetadata.html Sun May 15 21:40:24 2016 @@ -0,0 +1,160 @@ +<!DOCTYPE html> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE- 2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> + <head> + <title>Apache BookKeeper - BookKeeper Metadata Management</title> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <!-- Bootstrap --> + <link href="/css/bootstrap.min.css" rel="stylesheet"> + <link href="/css/bootstrap-responsive.min.css" rel="stylesheet"> + <link href="/css/styles.css" rel="stylesheet"> + </head> + <body> + <header class="navbar navbar-inverse navbar-static-top" role="banner"> + <div class="container"> + <div class="navbar-header hidden-xs hidden-sm"> + <a class="navbar-brand navbar-logo" href="/"><img class="img-responsive" src="/img/bookkeeper_blk40.png" alt="Bookkeeper Logo" /></a> + </div> + <div class="navbar-header"> + <button class="navbar-toggle collapsed" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/">Apache BookKeeper</a> + </div> + <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> + <ul class="nav navbar-nav"> + <li><a href="/releases.html">Download</a></li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/docs/master">Latest (master)</a></li> + <li><ul> + <li><a href="/docs/master/apidocs">Java API docs</a></li> + <li><a href="/docs/master/bookkeeperTutorial.html">Tutorial</a></li> + <li><a href="/docs/master/bookkeeperConfig.html">Admin guide</a></li> + </ul><li> + <li><a href="/docs/r4.4.0">Release 4.4.0</a></li> + <li class="divider"></li> + <li>Older releases</li> + <li><a href="/docs/r4.3.2">Release 4.3.2</a></li> + <li><a href="/docs/r4.3.1">Release 4.3.1</a></li> + <li><a href="/docs/r4.3.0">Release 4.3.0</a></li> + <li><a href="/docs/r4.2.4">Release 4.2.4</a></li> + <li><a href="/docs/r4.2.3">Release 4.2.3</a></li> + <li><a href="/docs/r4.2.2">Release 4.2.2</a></li> + <li><a href="/docs/r4.2.1">Release 4.2.1</a></li> + <li><a href="/docs/r4.2.0">Release 4.2.0</a></li> + <li><a href="/docs/r4.1.0">Release 4.1.0</a></li> + <li><a href="/docs/r4.0.0">Release 4.0.0</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Get Involved<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/lists.html">Mailing Lists</a></li> + <li><a href="/irc.html">IRC</a></li> + <li><a href="/svn.html">Version Control</a></li> + <li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER";>Issue Tracker</a></li> + </ul> + </li> + + <li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index";>Wiki</a></li> + <!--<li><a href="#">Hedwig</a></li>//--> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project Info<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/credits.html">Who are we?</a></li> + <li><a href="/bylaws.html">Bylaws</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li class="divider"></li> + <li><a href="/privacy.html">Privacy Policy</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsership</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + <script> + (function() { + var cx = '017580107654524087317:iqnsyimpydg'; + var gcse = document.createElement('script'); + gcse.type = 'text/javascript'; + gcse.async = true; + gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + + '//www.google.com/cse/cse.js?cx=' + cx; + var s = document.getElementsByTagName('script')[0]; + s.parentNode.insertBefore(gcse, s); + })(); + </script> + + <div class="navbar-form navbar-right visible-lg" id="googlebox"> + <gcse:searchbox-only></gcse:searchbox-only> + </div> + </nav> + </div> + </header> + <div class="container"> + + <h1>Metadata Management</h1> + +<p>There are two kinds of metadata needs to be managed in BookKeeper: one is the <i>list of available bookies</i>, which is used to track server availability (ZooKeeper is designed naturally for this); while the other is <i>ledger metadata</i>, which could be handle by different kinds of key/value storages efficiently with <i><span class="caps">CAS </span>(Compare And Set)</i> semantics.</p> + +<p><i>Ledger metadata</i> is handled by <i>LedgerManager</i> and can be plugged with various storage mediums.</p> + +<h2>Ledger Metadata Management</h2> + +<p>The operations on the metadata of a ledger are quite straightforward. They are:</p> + +<ul> +<li><code>createLedger</code>: create an new entry to store given ledger metadata. A unique id should be generated as the ledger id for the new ledger.</li> +<li><code>removeLedgerMetadata</code>: remove the entry of a ledger from metadata store. A <i>Version</i> object is provided to do conditional remove. If given <i>Version</i> object doesn't match current <i>Version</i> in metadata store, <i>MetadataVersionException</i> should be thrown to indicate version confliction. <i>NoSuchLedgerExistsException</i> should be returned if the ledger metadata entry doesn't exists.</li> +<li><code>readLedgerMetadata</code>: read the metadata of a ledger from metadata store. The new <i>version</i> should be set to the returned <i>LedgerMetadata</i> object. <i>NoSuchLedgerExistsException</i> should be returned if the entry of the ledger metadata doesn't exists.</li> +<li><code>writeLedgerMetadata</code>: update the metadata of a ledger matching the given <i>Version</i>. The update should be rejected and <i>MetadataVersionException</i> should be returned whe then given <i>Version</i> doesn't match the current <i>Version</i> in metadata store. <i>NoSuchLedgerExistsException</i> should be returned if the entry of the ledger metadata doesn't exists. The version of the <i>LedgerMetadata</i> object should be set to the new <i>Version</i> generated by applying this update.</li> +<li><code>asyncProcessLedgers</code>: loops through all existed ledgers in metadata store and applies a <i>Processor</i>. The <i>Processor</i> provided is executed for each ledger. If a failure happens during iteration, the iteration should be teminated and <i>final callback</i> triggered with failure. Otherwise, <i>final callback</i> is triggered after all ledgers are processed. No ordering nor transactional guarantees need to be provided for in the implementation of this interface.</li> +<li><code>getLedgerRanges</code>: return a list of ranges for ledgers in the metadata store. The ledger metadata itself does not need to be fetched. Only the ledger ids are needed. No ordering is required, but there must be no overlap between ledger ranges and each ledger range must be contain all the ledgers in the metadata store between the defined endpoint (i.e. a ledger range [x, y], all ledger ids larger or equal to x and smaller or equal to y should exist only in this range). <i>getLedgerRanges</i> is used in the <i>ScanAndCompare</i> gc algorithm.</li> +</ul> + +<h1>How to choose a metadata storage medium for BookKeeper.</h1> + +<p>From the interface, several requirements need to met before choosing a metadata storage medium for BookKeeper:</p> + +<ul> +<li><code>Check and Set (CAS)</code>: The ability to do strict update according to specific conditional. Etc, a specific version (ZooKeeper) and same content (HBase).</li> +<li><code>Optimized for Writes</code>: The metadata access pattern for BookKeeper is read first and continuous updates.</li> +<li><code>Optimized for Scans</code>: Scans are required for a <i>ScanAndCompare</i> gc algorithm.</li> +</ul> + +<p><i>ZooKeeper</i> is the default implemention for BookKeeper metadata management, <i>ZooKeeper</i> holds data in memory and provides filesystem-like namespace and also meets all the above requirements. <i>ZooKeeper</i> could meet most of usages for BookKeeper. However, if you application needs to manage millions of ledgers, a more scalable solution would be <i>HBase</i>, which also meet the above requirements, but it more complicated to set up.</p> + + </div> + <footer class="footer"> + <div class="container"> + <p class="text-muted">Copyright © 2014 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.<br/> + Apache BookKeeper, BookKeeper, Apache, Apache ZooKeeper, ZooKeeper, the Apache feather logo, and the Apache BookKeeper project logo are trademarks of The Apache Software Foundation.</p> + </div> + </footer> + + <script src="http://code.jquery.com/jquery.js";></script> + <script src="/js/bootstrap.min.js"></script> + </body> +</html> Added: websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperOverview.html ============================================================================== --- websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperOverview.html (added) +++ websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperOverview.html Sun May 15 21:40:24 2016 @@ -0,0 +1,331 @@ +<!DOCTYPE html> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE- 2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> + <head> + <title>Apache BookKeeper - BookKeeper overview</title> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <!-- Bootstrap --> + <link href="/css/bootstrap.min.css" rel="stylesheet"> + <link href="/css/bootstrap-responsive.min.css" rel="stylesheet"> + <link href="/css/styles.css" rel="stylesheet"> + </head> + <body> + <header class="navbar navbar-inverse navbar-static-top" role="banner"> + <div class="container"> + <div class="navbar-header hidden-xs hidden-sm"> + <a class="navbar-brand navbar-logo" href="/"><img class="img-responsive" src="/img/bookkeeper_blk40.png" alt="Bookkeeper Logo" /></a> + </div> + <div class="navbar-header"> + <button class="navbar-toggle collapsed" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/">Apache BookKeeper</a> + </div> + <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> + <ul class="nav navbar-nav"> + <li><a href="/releases.html">Download</a></li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/docs/master">Latest (master)</a></li> + <li><ul> + <li><a href="/docs/master/apidocs">Java API docs</a></li> + <li><a href="/docs/master/bookkeeperTutorial.html">Tutorial</a></li> + <li><a href="/docs/master/bookkeeperConfig.html">Admin guide</a></li> + </ul><li> + <li><a href="/docs/r4.4.0">Release 4.4.0</a></li> + <li class="divider"></li> + <li>Older releases</li> + <li><a href="/docs/r4.3.2">Release 4.3.2</a></li> + <li><a href="/docs/r4.3.1">Release 4.3.1</a></li> + <li><a href="/docs/r4.3.0">Release 4.3.0</a></li> + <li><a href="/docs/r4.2.4">Release 4.2.4</a></li> + <li><a href="/docs/r4.2.3">Release 4.2.3</a></li> + <li><a href="/docs/r4.2.2">Release 4.2.2</a></li> + <li><a href="/docs/r4.2.1">Release 4.2.1</a></li> + <li><a href="/docs/r4.2.0">Release 4.2.0</a></li> + <li><a href="/docs/r4.1.0">Release 4.1.0</a></li> + <li><a href="/docs/r4.0.0">Release 4.0.0</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Get Involved<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/lists.html">Mailing Lists</a></li> + <li><a href="/irc.html">IRC</a></li> + <li><a href="/svn.html">Version Control</a></li> + <li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER";>Issue Tracker</a></li> + </ul> + </li> + + <li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index";>Wiki</a></li> + <!--<li><a href="#">Hedwig</a></li>//--> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project Info<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/credits.html">Who are we?</a></li> + <li><a href="/bylaws.html">Bylaws</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li class="divider"></li> + <li><a href="/privacy.html">Privacy Policy</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsership</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + <script> + (function() { + var cx = '017580107654524087317:iqnsyimpydg'; + var gcse = document.createElement('script'); + gcse.type = 'text/javascript'; + gcse.async = true; + gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + + '//www.google.com/cse/cse.js?cx=' + cx; + var s = document.getElementsByTagName('script')[0]; + s.parentNode.insertBefore(gcse, s); + })(); + </script> + + <div class="navbar-form navbar-right visible-lg" id="googlebox"> + <gcse:searchbox-only></gcse:searchbox-only> + </div> + </nav> + </div> + </header> + <div class="container"> + + <h1>Abstract</h1> + +<p>This guide contains detailed information about using BookKeeper for logging. It discusses the basic operations BookKeeper supports, and how to create logs and perform basic read and write operations on these logs.</p> + +<h1>BookKeeper introduction</h1> + +<p>BookKeeper is a replicated service to reliably log streams of records. In BookKeeper, servers are "bookies", log streams are "ledgers", and each unit of a log (aka record) is a "ledger entry". BookKeeper is designed to be reliable; bookies, the servers that store ledgers, can crash, corrupt data, discard data, but as long as there are enough bookies behaving correctly the service as a whole behaves correctly. </p> + +<p>The initial motivation for BookKeeper comes from the namenode of <span class="caps">HDFS.</span> Namenodes have to log operations in a reliable fashion so that recovery is possible in the case of crashes. We have found the applications for BookKeeper extend far beyond <span class="caps">HDFS, </span>however. Essentially, any application that requires an append storage can replace their implementations with BookKeeper. BookKeeper has the advantage of writing efficiently, replicating for fault tolerance, and scaling throughput with the number of servers through striping. </p> + +<p>At a high level, a bookkeeper client receives entries from a client application and stores it to sets of bookies, and there are a few advantages in having such a service: </p> + +<ul> +<li>We can use hardware that is optimized for such a service. We currently believe that such a system has to be optimized only for disk I/O; </li> +<li>We can have a pool of servers implementing such a log system, and shared among a number of servers; </li> +<li>We can have a higher degree of replication with such a pool, which makes sense if the hardware necessary for it is cheaper compared to the one the application uses. </li> +</ul> + + +<h1>In slightly more detail...</h1> + +<p>BookKeeper implements highly available logs, and it has been designed with write-ahead logging in mind. Besides high availability due to the replicated nature of the service, it provides high throughput due to striping. As we write entries in a subset of bookies of an ensemble and rotate writes across available quorums, we are able to increase throughput with the number of servers for both reads and writes. Scalability is a property that is possible to achieve in this case due to the use of quorums. Other replication techniques, such as state-machine replication, do not enable such a property. </p> + +<p>An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper. </p> + +<p>Note that currently this recovery procedure is executed automatically upon trying to open a ledger and no explicit action is necessary. Although two clients may try to recover a ledger concurrently, only one will succeed, the first one that is able to create the close znode for the ledger. </p> + +<h1>Bookkeeper elements and concepts</h1> + +<p>BookKeeper uses four basic elements: </p> + +<ul> +<li> <em>Ledger</em> : A ledger is a sequence of entries, and each entry is a sequence of bytes. Entries are written sequentially to a ledger and at most once. Consequently, ledgers have an append-only semantics; </li> +<li> <em>BookKeeper client</em> : A client runs along with a BookKeeper application, and it enables applications to execute operations on ledgers, such as creating a ledger and writing to it; </li> +<li> <em>Bookie</em> : A bookie is a BookKeeper storage server. Bookies store the content of ledgers. For any given ledger L, we call an <em>ensemble</em> the group of bookies storing the content of L. For performance, we store on each bookie of an ensemble only a fragment of a ledger. That is, we stripe when writing entries to a ledger such that each entry is written to sub-group of bookies of the ensemble. </li> +<li> <em>Metadata storage service</em> : BookKeeper requires a metadata storage service to store information related to ledgers and available bookies. We currently use ZooKeeper for such a task. </li> +</ul> + + +<h1>Bookkeeper initial design</h1> + +<p>A set of bookies implements BookKeeper, and we use a quorum-based protocol to replicate data across the bookies. There are basically two operations to an existing ledger: read and append. Here is the complete <span class="caps">API </span>list (more detail <a href="./bookkeeperProgrammer.html">here</a>):</p> + +<ul> +<li>Create ledger: creates a new empty ledger; </li> +<li>Open ledger: opens an existing ledger for reading; </li> +<li>Add entry: adds a record to a ledger either synchronously or asynchronously; </li> +<li>Read entries: reads a sequence of entries from a ledger either synchronously or asynchronously </li> +</ul> + + +<p>There is only a single client that can write to a ledger. Once that ledger is closed or the client fails, no more entries can be added. (We take advantage of this behavior to provide our strong guarantees.) There will not be gaps in the ledger. Fingers get broken, people get roughed up or end up in prison when books are manipulated, so there is no deleting or changing of entries. </p> + +<p><img src="images/bk-overview.jpg" alt="" /> +p. A simple use of BookKeeper is to implement a write-ahead transaction log. A server maintains an in-memory data structure (with periodic snapshots for example) and logs changes to that structure before it applies the change. The application server creates a ledger at startup and store the ledger id and password in a well known place (ZooKeeper maybe). When it needs to make a change, the server adds an entry with the change information to a ledger and apply the change when BookKeeper adds the entry successfully. The server can even use asyncAddEntry to queue up many changes for high change throughput. BookKeeper meticulously logs the changes in order and call the completion functions in order.</p> + +<p>When the application server dies, a backup server will come online, get the last snapshot and then it will open the ledger of the old server and read all the entries from the time the snapshot was taken. (Since it doesn't know the last entry number it will use <span class="caps">MAX</span>_INTEGER). Once all the entries have been processed, it will close the ledger and start a new one for its use. </p> + +<p>A client library takes care of communicating with bookies and managing entry numbers. An entry has the following fields: </p> + +<table><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td>Ledger number</td><td>long</td><td>The id of the ledger of this entry</td></tr><tr><td>Entry number</td><td>long</td><td>The id of this entry</td></tr><tr><td>last confirmed ( <em>LC</em> )</td><td>long</td><td>id of the last recorded entry</td></tr><tr><td>data</td><td>byte[]</td><td>the entry data (supplied by application)</td></tr><tr><td>authentication code</td><td>byte[]</td><td>Message authentication code that includes all other fields of the entry</td></tr></table> + + +<p>The client library generates a ledger entry. None of the fields are modified by the bookies and only the first three fields are interpreted by the bookies. </p> + +<p>To add to a ledger, the client generates the entry above using the ledger number. The entry number will be one more than the last entry generated. The <em>LC</em> field contains the last entry that has been successfully recorded by BookKeeper. If the client writes entries one at a time, <em>LC</em> is the last entry id. But, if the client is using asyncAddEntry, there may be many entries in flight. An entry is considered recorded when both of the following conditions are met: </p> + +<ul> +<li>the entry has been accepted by a quorum of bookies </li> +<li>all entries with a lower entry id have been accepted by a quorum of bookies </li> +</ul> + + +<p> <em>LC</em> seems mysterious right now, but it is too early to explain how we use it; just smile and move on. </p> + +<p>Once all the other fields have been field in, the client generates an authentication code with all of the previous fields. The entry is then sent to a quorum of bookies to be recorded. Any failures will result in the entry being sent to a new quorum of bookies. </p> + +<p>To read, the client library initially contacts a bookie and starts requesting entries. If an entry is missing or invalid (a bad <span class="caps">MAC </span>for example), the client will make a request to a different bookie. By using quorum writes, as long as enough bookies are up we are guaranteed to eventually be able to read an entry. </p> + +<h1>Bookkeeper metadata management</h1> + +<p>There are some meta data that needs to be made available to BookKeeper clients: </p> + +<ul> +<li>The available bookies; </li> +<li>The list of ledgers; </li> +<li>The list of bookies that have been used for a given ledger; </li> +<li>The last entry of a ledger; </li> +</ul> + + +<p>We maintain this information in ZooKeeper. Bookies use ephemeral nodes to indicate their availability. Clients use znodes to track ledger creation and deletion and also to know the end of the ledger and the bookies that were used to store the ledger. Bookies also watch the ledger list so that they can cleanup ledgers that get deleted. </p> + +<h1>Closing out ledgers</h1> + +<p>The process of closing out the ledger and finding the last entry is difficult due to the durability guarantees of BookKeeper: </p> + +<ul> +<li>If an entry has been successfully recorded, it must be readable. </li> +<li>If an entry is read once, it must always be available to be read. </li> +</ul> + + +<p>If the ledger was closed gracefully, ZooKeeper will have the last entry and everything will work well. But, if the BookKeeper client that was writing the ledger dies, there is some recovery that needs to take place. </p> + +<p>The problematic entries are the ones at the end of the ledger. There can be entries in flight when a BookKeeper client dies. If the entry only gets to one bookie, the entry should not be readable since the entry will disappear if that bookie fails. If the entry is only on one bookie, that doesn't mean that the entry has not been recorded successfully; the other bookies that recorded the entry might have failed. </p> + +<p>The trick to making everything work is to have a correct idea of a last entry. We do it in roughly three steps: </p> + +<ol> +<li>Find the entry with the highest last recorded entry, <em>LC</em> ; </li> +<li>Find the highest consecutively recorded entry, <em>LR</em> ; </li> +<li>Make sure that all entries between <em>LC</em> and <em>LR</em> are on a quorum of bookies; </li> +</ol> + +<h1>Data Management in Bookies</h1> + +<p>This section gives an overview of how a bookie manages its ledger fragments. </p> + +<h2>Basic</h2> + +<p>Bookies manage data in a log-structured way, which is implemented using three kind of files:</p> + +<ul> +<li><em>Journal</em> : A journal file contains the BookKeeper transaction logs. Before any update takes place, a bookie ensures that a transaction describing the update is written to non-volatile storage. A new journal file is created once the bookie starts or the older journal file reaches the journal file size threshold.</li> +<li><em>Entry Log</em> : An entry log file manages the written entries received from BookKeeper clients. Entries from different ledgers are aggregated and written sequentially, while their offsets are kept as pointers in <em>LedgerCache</em> for fast lookup. A new entry log file is created once the bookie starts or the older entry log file reaches the entry log size threshold. Old entry log files are removed by the <em>Garbage Collector Thread</em> once they are not associated with any active ledger.</li> +<li><em>Index File</em> : An index file is created for each ledger, which comprises a header and several fixed-length index pages, recording the offsets of data stored in entry log files. </li> +</ul> + +<p>Since updating index files would introduce random disk I/O, for performance consideration, index files are updated lazily by a <em>Sync Thread</em> running in the background. Before index pages are persisted to disk, they are gathered in <em>LedgerCache</em> for lookup.</p> + +<ul> +<li><em>LedgerCache</em> : A memory pool caches ledger index pages, which more efficiently manage disk head scheduling.</li> +</ul> + +<h2>Add Entry</h2> + +<p>When a bookie receives entries from clients to be written, these entries will go through the following steps to be persisted to disk:</p> + +<ol> +<li>Append the entry in <em>Entry Log</em>, return its position { logId , offset } ;</li> +<li>Update the index of this entry in <em>Ledger Cache</em> ;</li> +<li>Append a transaction corresponding to this entry update in <em>Journal</em> ;</li> +<li>Respond to BookKeeper client ;</li> +</ol> + +<ul> +<li>For performance reasons, <em>Entry Log</em> buffers entries in memory and commit them in batches, while <em>Ledger Cache</em> holds index pages in memory and flushes them lazily. We will discuss data flush and how to ensure data integrity in the following section 'Data Flush'.</li> +</ul> + +<h2>Data Flush</h2> + +<p>Ledger index pages are flushed to index files in the following two cases:</p> + +<ol> +<li><em>LedgerCache</em> memory reaches its limit. There is no more space available to hold newer index pages. Dirty index pages will be evicted from <em>LedgerCache</em> and persisted to index files.</li> +<li>A background thread <em>Sync Thread</em> is responsible for flushing index pages from <em>LedgerCache</em> to index files periodically.</li> +</ol> + +<p>Besides flushing index pages, <em>Sync Thread</em> is responsible for rolling journal files in case that journal files use too much disk space. </p> + +<p>The data flush flow in <em>Sync Thread</em> is as follows:</p> + +<ol> +<li>Records a <em>LastLogMark</em> in memory. The <em>LastLogMark</em> contains two parts: first one is <em>txnLogId</em> (file id of a journal) and the second one is <em>txnLogPos</em> (offset in a journal). The <em>LastLogMark</em> indicates that those entries before it have been persisted to both index and entry log files.</li> +<li>Flushes dirty index pages from <em>LedgerCache</em> to index file, and flushes entry log files to ensure all buffered entries in entry log files are persisted to disk.<ol> +<li>Ideally, a bookie just needs to flush index pages and entry log files that contains entries before <em>LastLogMark</em>. There is no such information in <em>LedgerCache</em> and <em>Entry Log</em> mapping to journal files, though. Consequently, the thread flushes <em>LedgerCache</em> and <em>Entry Log</em> entirely here, and may flush entries after the <em>LastLogMark</em>. Flushing more is not a problem, though, just redundant.</li> +</ol> +</li> +<li>Persists <em>LastLogMark</em> to disk, which means entries added before <em>LastLogMark</em> whose entry data and index page were also persisted to disk. It is the time to safely remove journal files created earlier than <em>txnLogId</em>.<ol> +<li>If the bookie has crashed before persisting <em>LastLogMark</em> to disk, it still has journal files containing entries for which index pages may not have been persisted. Consequently, when this bookie restarts, it inspects journal files to restore those entries; data isn't lost.</li> +</ol></li> +</ol> + +<p>Using the above data flush mechanism, it is safe for the <em>Sync Thread</em> to skip data flushing when the bookie shuts down. However, in <em>Entry Logger</em>, it uses <em>BufferedChannel</em> to write entries in batches and there might be data buffered in <em>BufferedChannel</em> upon a shut down. The bookie needs to ensure <em>Entry Logger</em> flushes its buffered data during shutting down. Otherwise, <em>Entry Log</em> files become corrupted with partial entries.</p> + +<p>As described above, <em>EntryLogger#flush</em> is invoked in the following two cases:<br /> +* in <em>Sync Thread</em> : used to ensure entries added before <em>LastLogMark</em> are persisted to disk.<br /> +* in <em>ShutDown</em> : used to ensure its buffered data persisted to disk to avoid data corruption with partial entries.</p> + +<h2>Data Compaction</h2> + +<p>In bookie server, entries of different ledgers are interleaved in entry log files. A bookie server runs a <em>Garbage Collector</em> thread to delete un-associated entry log files to reclaim disk space. If a given entry log file contains entries from a ledger that has not been deleted, then the entry log file would never be removed and the occupied disk space never reclaimed. In order to avoid such a case, a bookie server compacts entry log files in <em>Garbage Collector</em> thread to reclaim disk space.</p> + +<p>There are two kinds of compaction running with different frequency, which are <em>Minor Compaction</em> and <em>Major Compaction</em>. The differences of <em>Minor Compaction</em> and <em>Major Compaction</em> are just their threshold value and compaction interval.</p> + +<ol> +<li><em>Threshold</em> : Size percentage of an entry log file occupied by those undeleted ledgers. Default minor compaction threshold is 0.2, while major compaction threshold is 0.8.</li> +<li><em>Interval</em> : How long to run the compaction. Default minor compaction is 1 hour, while major compaction threshold is 1 day.</li> +</ol> + +<p><span class="caps">NOTE</span>: if either <em>Threshold</em> or <em>Interval</em> is set to less than or equal to zero, then compaction is disabled.</p> + +<p>The data compaction flow in <em>Garbage Collector Thread</em> is as follows:</p> + +<ol> +<li><em>Garbage Collector</em> thread scans entry log files to get their entry log metadata, which records a list of ledgers comprising an entry log and their corresponding percentages.</li> +<li>With the normal garbage collection flow, once the bookie determines that a ledger has been deleted, the ledger will be removed from the entry log metadata and the size of the entry log reduced.</li> +<li>If the remaining size of an entry log file reaches a specified threshold, the entries of active ledgers in the entry log will be copied to a new entry log file.</li> +<li>Once all valid entries have been copied, the old entry log file is deleted.</li> +</ol> + + </div> + <footer class="footer"> + <div class="container"> + <p class="text-muted">Copyright © 2014 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.<br/> + Apache BookKeeper, BookKeeper, Apache, Apache ZooKeeper, ZooKeeper, the Apache feather logo, and the Apache BookKeeper project logo are trademarks of The Apache Software Foundation.</p> + </div> + </footer> + + <script src="http://code.jquery.com/jquery.js";></script> + <script src="/js/bootstrap.min.js"></script> + </body> +</html> Added: websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperProgrammer.html ============================================================================== --- websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperProgrammer.html (added) +++ websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperProgrammer.html Sun May 15 21:40:24 2016 @@ -0,0 +1,247 @@ +<!DOCTYPE html> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE- 2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> + <head> + <title>Apache BookKeeper - BookKeeper Getting Started Guide</title> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <!-- Bootstrap --> + <link href="/css/bootstrap.min.css" rel="stylesheet"> + <link href="/css/bootstrap-responsive.min.css" rel="stylesheet"> + <link href="/css/styles.css" rel="stylesheet"> + </head> + <body> + <header class="navbar navbar-inverse navbar-static-top" role="banner"> + <div class="container"> + <div class="navbar-header hidden-xs hidden-sm"> + <a class="navbar-brand navbar-logo" href="/"><img class="img-responsive" src="/img/bookkeeper_blk40.png" alt="Bookkeeper Logo" /></a> + </div> + <div class="navbar-header"> + <button class="navbar-toggle collapsed" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/">Apache BookKeeper</a> + </div> + <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> + <ul class="nav navbar-nav"> + <li><a href="/releases.html">Download</a></li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/docs/master">Latest (master)</a></li> + <li><ul> + <li><a href="/docs/master/apidocs">Java API docs</a></li> + <li><a href="/docs/master/bookkeeperTutorial.html">Tutorial</a></li> + <li><a href="/docs/master/bookkeeperConfig.html">Admin guide</a></li> + </ul><li> + <li><a href="/docs/r4.4.0">Release 4.4.0</a></li> + <li class="divider"></li> + <li>Older releases</li> + <li><a href="/docs/r4.3.2">Release 4.3.2</a></li> + <li><a href="/docs/r4.3.1">Release 4.3.1</a></li> + <li><a href="/docs/r4.3.0">Release 4.3.0</a></li> + <li><a href="/docs/r4.2.4">Release 4.2.4</a></li> + <li><a href="/docs/r4.2.3">Release 4.2.3</a></li> + <li><a href="/docs/r4.2.2">Release 4.2.2</a></li> + <li><a href="/docs/r4.2.1">Release 4.2.1</a></li> + <li><a href="/docs/r4.2.0">Release 4.2.0</a></li> + <li><a href="/docs/r4.1.0">Release 4.1.0</a></li> + <li><a href="/docs/r4.0.0">Release 4.0.0</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Get Involved<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/lists.html">Mailing Lists</a></li> + <li><a href="/irc.html">IRC</a></li> + <li><a href="/svn.html">Version Control</a></li> + <li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER";>Issue Tracker</a></li> + </ul> + </li> + + <li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index";>Wiki</a></li> + <!--<li><a href="#">Hedwig</a></li>//--> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project Info<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/credits.html">Who are we?</a></li> + <li><a href="/bylaws.html">Bylaws</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li class="divider"></li> + <li><a href="/privacy.html">Privacy Policy</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsership</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + <script> + (function() { + var cx = '017580107654524087317:iqnsyimpydg'; + var gcse = document.createElement('script'); + gcse.type = 'text/javascript'; + gcse.async = true; + gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + + '//www.google.com/cse/cse.js?cx=' + cx; + var s = document.getElementsByTagName('script')[0]; + s.parentNode.insertBefore(gcse, s); + })(); + </script> + + <div class="navbar-form navbar-right visible-lg" id="googlebox"> + <gcse:searchbox-only></gcse:searchbox-only> + </div> + </nav> + </div> + </header> + <div class="container"> + + <h1>Abstract</h1> + +<p>This guide contains detailed information about using BookKeeper for write ahead logging. It discusses the basic operations BookKeeper supports, and how to create logs and perform basic read and write operations on these logs. The main classes used by BookKeeper client are <a href="./apidocs/org/apache/bookkeeper/client/BookKeeper.html">BookKeeper</a> and <a href="./apidocs/org/apache/bookkeeper/client/LedgerHandle.html">LedgerHandle</a>. </p> + +<p>BookKeeper is the main client used to create, open and delete ledgers. A ledger is a log file in BookKeeper, which contains a sequence of entries. Only the client which creates a ledger can write to it. A LedgerHandle represents the ledger to the client, and allows the client to read and write entries. When the client is finished writing they can close the LedgerHandle. Once a ledger has been closed, all client who read from it are guaranteed to read the exact same entries in the exact same order. All methods of BookKeeper and LedgerHandle have synchronous and asynchronous versions. Internally the synchronous versions are implemented using the asynchronous.</p> + +<h1> Instantiating BookKeeper</h1> + +<p>To create a BookKeeper client, you need to create a configuration object and set the address of the ZooKeeper ensemble in use. For example, if you were using <code>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</code> as your ensemble, you would create the BookKeeper client as follows.</p> + + + +<pre><code> +ClientConfiguration conf = new ClientConfiguration(); +conf.setZkServers("zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181"); + +BookKeeper client = new BookKeeper(conf); +</code></pre> + + + +<p>It is important to close the client once you are finished working with it. The set calls on ClientConfiguration are chainable, so instead of putting a set* call on a new line as above, it is possible to make a number of calls on the one line. For example;</p> + + + +<pre><code> +ClientConfiguration conf = new ClientConfiguration().setZkServers("localhost:2181").setZkTimeout(5000); +</code></pre> + + + +<p>There is also a useful shortcut constructor which allows you to pass the zookeeper ensemble string directly to BookKeeper.</p> + + +<pre><code> +BookKeeper client = new BookKeeper("localhost:2181"); +</code></pre> + + + +<p>See <a href="./apidocs/org/apache/bookkeeper/client/BookKeeper.html">BookKeeper</a> for the full api.</p> + + +<h1> Creating a ledger</h1> + +<p>Before writing entries to BookKeeper, it is necessary to create a ledger. Before creating the ledger you must decide the ensemble size and the quorum size. </p> + +<p>The ensemble size is the number of Bookies over which entries will be striped. The quorum size is the number of bookies which an entry will be written to. Striping is done in a round robin fashion. For example, if you have an ensemble size of 3 (consisting of bk1, bk2 & bk3), and a quorum of 2, entry 1 will be written to bk1 & bk2, entry 2 will be written to bk2 & bk3, entry 3 will be written to bk3 & bk1 and so on.</p> + +<p>Ledgers are also created with a digest type and password. The digest type is used to generate a checksum so that when reading entries we can ensure that the content is the same as what was written. The password is used as an access control mechanism.</p> + +<p>To create a ledger, with ensemble size 3, quorum size 2, using a <span class="caps">CRC </span>to checksum and "foobar" as the password, do the following:</p> + + + +<pre><code> +LedgerHandle lh = client.createLedger(3, 2, DigestType.CRC32, "foobar"); +</code></pre> + + + +<p>You can now write to this ledger handle. As you probably plan to read the ledger at some stage, now is a good time to store the id of the ledger somewhere. The ledger id is a long, and can be obtained with <code>lh.getId()</code>.</p> + +<h1> Adding entries to a ledger</h1> + +<p>Once you have obtained a ledger handle, you can start adding entries to it. Entries are simply arrays of bytes. As such, adding entries to the ledger is rather simple.</p> + + + +<pre><code> +lh.addEntry("Hello World!".getBytes()); +</code></pre> + + + +<h1> Closing a ledger</h1> + +<p>Once a client is done writing, it can closes the ledger. Closing the ledger is a very important step in BookKeeper, as once a ledger is closed, all reading clients are guaranteed to read the same sequence of entries in the same order. Closing takes no parameters. </p> + + + +<pre><code> +lh.close(); +</code></pre> + + + +<h1> Opening a ledger</h1> + +<p>To read from a ledger, a client must open it first. To open a ledger you must know its <span class="caps">ID, </span>which digest type was used when creating it, and its password. To open the ledger we created above, assuming it has ID 1;</p> + + + +<pre><code> +LedgerHandle lh2 = client.openLedger(1, DigestType.CRC32, "foobar"); +</code></pre> + + + +<p>You can now read entries from the ledger. Any attempt to write to this handle will throw an exception.</p> + +<blockquote><p><span class="caps">NOTE</span>: Opening a ledger, which another client already has open for writing will prevent that client from writing any new entries to it. If you do not wish this to happen, you should use the openLedgerNoRecovery method. However, keep in mind that without recovery, you lose the guarantees of what entries are in the ledger. You should only use openLedgerNoRecovery if you know what you are doing.</p></blockquote> + +<h1>Reading entries from a ledger</h1> + +<p>Now that you have an open ledger, you can read entries from it. You can use <code>getLastAddConfirmed</code> to get the id of the last entry in the ledger.</p> + + + +<pre><code> +long lastEntry = lh2.getLastAddConfirmed(); +Enumeration<LedgerEntry> entries = lh2.readEntries(0, 9); +while (entries.hasMoreElements()) { + byte[] bytes = entries.nextElement().getEntry(); + System.out.println(new String(bytes)); +} +</code></pre> + + </div> + <footer class="footer"> + <div class="container"> + <p class="text-muted">Copyright © 2014 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.<br/> + Apache BookKeeper, BookKeeper, Apache, Apache ZooKeeper, ZooKeeper, the Apache feather logo, and the Apache BookKeeper project logo are trademarks of The Apache Software Foundation.</p> + </div> + </footer> + + <script src="http://code.jquery.com/jquery.js";></script> + <script src="/js/bootstrap.min.js"></script> + </body> +</html> Added: websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperProtocol.html ============================================================================== --- websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperProtocol.html (added) +++ websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperProtocol.html Sun May 15 21:40:24 2016 @@ -0,0 +1,219 @@ +<!DOCTYPE html> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE- 2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> + <head> + <title>Apache BookKeeper - BookKeeper Replication Protocol</title> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <!-- Bootstrap --> + <link href="/css/bootstrap.min.css" rel="stylesheet"> + <link href="/css/bootstrap-responsive.min.css" rel="stylesheet"> + <link href="/css/styles.css" rel="stylesheet"> + </head> + <body> + <header class="navbar navbar-inverse navbar-static-top" role="banner"> + <div class="container"> + <div class="navbar-header hidden-xs hidden-sm"> + <a class="navbar-brand navbar-logo" href="/"><img class="img-responsive" src="/img/bookkeeper_blk40.png" alt="Bookkeeper Logo" /></a> + </div> + <div class="navbar-header"> + <button class="navbar-toggle collapsed" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/">Apache BookKeeper</a> + </div> + <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> + <ul class="nav navbar-nav"> + <li><a href="/releases.html">Download</a></li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/docs/master">Latest (master)</a></li> + <li><ul> + <li><a href="/docs/master/apidocs">Java API docs</a></li> + <li><a href="/docs/master/bookkeeperTutorial.html">Tutorial</a></li> + <li><a href="/docs/master/bookkeeperConfig.html">Admin guide</a></li> + </ul><li> + <li><a href="/docs/r4.4.0">Release 4.4.0</a></li> + <li class="divider"></li> + <li>Older releases</li> + <li><a href="/docs/r4.3.2">Release 4.3.2</a></li> + <li><a href="/docs/r4.3.1">Release 4.3.1</a></li> + <li><a href="/docs/r4.3.0">Release 4.3.0</a></li> + <li><a href="/docs/r4.2.4">Release 4.2.4</a></li> + <li><a href="/docs/r4.2.3">Release 4.2.3</a></li> + <li><a href="/docs/r4.2.2">Release 4.2.2</a></li> + <li><a href="/docs/r4.2.1">Release 4.2.1</a></li> + <li><a href="/docs/r4.2.0">Release 4.2.0</a></li> + <li><a href="/docs/r4.1.0">Release 4.1.0</a></li> + <li><a href="/docs/r4.0.0">Release 4.0.0</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Get Involved<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/lists.html">Mailing Lists</a></li> + <li><a href="/irc.html">IRC</a></li> + <li><a href="/svn.html">Version Control</a></li> + <li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER";>Issue Tracker</a></li> + </ul> + </li> + + <li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index";>Wiki</a></li> + <!--<li><a href="#">Hedwig</a></li>//--> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project Info<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/credits.html">Who are we?</a></li> + <li><a href="/bylaws.html">Bylaws</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li class="divider"></li> + <li><a href="/privacy.html">Privacy Policy</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsership</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + <script> + (function() { + var cx = '017580107654524087317:iqnsyimpydg'; + var gcse = document.createElement('script'); + gcse.type = 'text/javascript'; + gcse.async = true; + gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + + '//www.google.com/cse/cse.js?cx=' + cx; + var s = document.getElementsByTagName('script')[0]; + s.parentNode.insertBefore(gcse, s); + })(); + </script> + + <div class="navbar-form navbar-right visible-lg" id="googlebox"> + <gcse:searchbox-only></gcse:searchbox-only> + </div> + </nav> + </div> + </header> + <div class="container"> + + <p>This documents describes the bookkeeper replication protocol, and the guarantees it gives. It assumes you have a general idea about leader election and log replication and how you can use these in your system. If not, have a look at the bookkeeper <a href="./bookkeeperTutorial.html">tutorial</a> first.</p> + +<h1>Ledgers</h1> + +<p>A ledger is the basic building block in Bookkeeper. All guarantees we provide are on ledgers. A replicated log is composed of an ordered list of ledgers. See <a href="./bookkeeperLedgers2Logs.html">From Ledgers to Logs</a> on how to build a replicated log from ledgers.</p> + +<p>Ledgers are composed of metadata and entries. The metadata is stored in a datastore which provides a compare-and-swap operation (generally ZooKeeper). Entries are stored on storage nodes known as bookies.</p> + +<p>A ledger has a single write and multiple readers (SWMR).</p> + +<p>A ledger's metadata contains:</p> + +<ul> +<li>id: a 64bit integer, unique within the system</li> +<li>ensemble size (E): the number of nodes the ledger is stored on.</li> +<li>write quorum size (Q<sub>w</sub>): the number of nodes each entry is written. In effect, the max replication for an entry.</li> +<li>ack quorum size (Q<sub>a</sub>): the number of nodes a entry must be acknowledge on. In effect, the min replication for an entry.</li> +<li>state: <span class="caps">OPEN, CLOSED </span>or IN_RECOVERY</li> +<li>last entry: the last entry in the ledger, or <span class="caps">NULL </span>if state != <span class="caps">CLOSED</span></li> +<li>one or more fragments, which each consist of:<ul> +<li>First entry of fragment, list of bookies for fragment</li> +</ul></li> +</ul> + +<p>When creating the ledger, the following invariant must hold.</p> + +<p> E >= Q<sub>w</sub> >= Q<sub>a</sub></p> + +<h4>Ensembles</h4> + +<p>When the ledger is created, E bookies are chosen for the entries of that ledger. The bookies are the initial ensemble of the ledger. A ledger can have multiple ensembles, but an entry has only one ensemble. Changes in the ensemble, involves a new fragment being added to the ledger.</p> + +<p>Take the following example. In this ledger, with ensemble size of 3, there are two fragments and thus two ensembles, one starting at entry 0, the second at entry 12. The second ensemble differs from the first only by its first element. This could be because bookie1 has failed and therefore had to be replaced.</p> + +<table class="table table-bordered table-hover" cellspacing="0"><tr><th>FirstEntry</th><th>Bookies</th></tr><tr><td>0</td><td><span class="caps">B1, B2,</span> B3</td></tr><tr><td>12</td><td><span class="caps">B4, B2,</span> B3</td></tr></table> + +<h4>Write Quorums</h4> + +<p>Each entry in the log is written to Q<sub>w</sub> nodes. This is considered the write quorum for that entry. The write quorum is the subsequence of the ensemble, Q<sub>w</sub> in length, and starting at the bookie at index (entryid % E).</p> + +<p>For example, in a ledger of E = 4, Q<sub>w</sub> = 3 & Q<sub>a</sub> = 2, with an ensemble consisting of <span class="caps">B1, B2,</span> B3 & <span class="caps">B4, </span>the write quorums for the first 6 entries will be.</p> + +<table class="table table-bordered table-hover" cellspacing="0"><tr><th>Entry</th><th>Write quorum</th></tr><tr><td>0</td><td><span class="caps">B1, B2,</span> B3</td></tr><tr><td>1</td><td><span class="caps">B2, B3,</span> B4</td></tr><tr><td>2</td><td><span class="caps">B3, B4,</span> B1</td></tr><tr><td>3</td><td><span class="caps">B4, B1,</span> B2</td></tr><tr><td>4</td><td><span class="caps">B1, B2,</span> B3</td></tr><tr><td>5</td><td><span class="caps">B2, B3,</span> B4</td></tr></table> + +<p>There are only E distinct write quorums in any ensemble. If Q<sub>w</sub> = Q<sub>a</sub>, then there is only one, as no striping occurs. </p> + +<h4>Ack Quorums</h4> + +<p>The ack quorum for an entry is any subset of the write quorum of size Q<sub>a</sub>. If Q<sub>a</sub> bookies acknowledge an entry, it means it has been fully replicated.</p> + +<h4>Guarantees</h4> + +<p>The system can tolerate Q<sub>a</sub> - 1 failures without data loss.</p> + +<p>Bookkeeper guarantees that:<br /> + 1. all updates to a ledger will be read in the same order as they were written <br /> + 2. all clients will read the same sequence of updates from the ledger</p> + +<h1>Writing to the ledger</h1> + +<p>When an entry is written to a ledger, it is assigned an entry id the write quorum is calculated. As there is only a single writer, ensuring that entry ids are sequential is trivial. A bookie acknowledges a write once it has been persisted to disk and is therefore durable. Once Q<sub>a</sub> bookies from the write quorum acknowledge the write, the write is acknowledged to the client, but only if all entries with lower entry ids in the ledger have already been acknowledged to the client.</p> + +<p>The entry written contains the ledger id, the entry id, the last add confirmed and the payload. The last add confirmed is the last entry which had been acknowledged to the client when this entry was written. Sending this with the entry speeds up recovery of the ledger in the case that the writer crashes.</p> + +<p>Another client can also read entries in the ledger up as far as the last add confirmed, as we guarantee that all entries thus far have been replicated on Q<sub>a</sub> nodes, and therefore all future readers will be able to also read it. However, to read like this, the ledger should be opened with a non-fencing open. Otherwise, it would kill the writer.</p> + +<p>If a node fails to acknowledge a write, the writer will create a new ensemble by replacing the failed node in the current ensemble. It creates a new fragment with this ensemble, starting from the first message that has not been acknowledged to the client. Creating the new fragment involves making a <span class="caps">CAS </span>write to the metadata. If the <span class="caps">CAS </span>write fails, someone else has modified something in the ledger metadata. This concurrent modification could have been caused by recovery or rereplication<sup class="footnote"><a href="#fn1">1</a></sup>. We reread the metadata. If the state of the ledger is no longer <span class="caps">OPEN, </span>we send an error to the client for any outstanding writes. Otherwise, we try to replace the failed node again.</p> + +<h1>Closing a ledger as a writer</h1> + +<p>Closing a ledger is straight forward for a writer. The writer makes a <span class="caps">CAS </span>write to the metadata, changing the state to <span class="caps">CLOSED, </span>and setting the last entry of the ledger to the last entry which we have acknowledged to the client.</p> + +<p>If the <span class="caps">CAS </span>write fails, it means someone else has modified the metadata. We reread the metadata, and retry closing as long as the state of the ledger is still <span class="caps">OPEN.</span> If the state is IN_RECOVERY we send an error to the client. If the state is <span class="caps">CLOSED </span>and the last entry is the same as the last entry we have acknowledged to the client, we complete the close operation successfully. If the last entry is different to what we have acknowledged to the client, we send an error to the client.</p> + +<h1>Closing a ledger as a reader</h1> + +<p>A reader can also force a ledger to close. Forcing the ledger to close will prevent any writer from adding new entries to the ledger. This is called <strong>Fencing</strong>. This can occur when a writer has crashed or has become unavailable, and a new writer wants to take over writing to the log. The new writer must ensure that it has seen all updates from the previous writer, and prevent the previous writer from making any new updates before making any updates of its own.</p> + +<p>To recover a ledger, we first update the state in the metadata to IN_RECOVERY. We then send a fence message to all the bookies in the last fragment of the ledger. When a bookie receives a fence message for a ledger, the fenced state of the ledger is persisted to disk. Once we receive a response from at least (Q<sub>w</sub>-Q<sub>a</sub>)+1 bookies from each write quorum in the ensemble, the ledger is fenced.</p> + +<p>By ensuring we have received a response from at last (Q<sub>w</sub>-Q<sub>a</sub>)+1 bookies in each write quorum, we ensure that, if the old writer is alive and tries to add a new entry there will be no write quorum in which Q<sub>a</sub> bookies will accept the write. If the old writer tries to update the ensemble, it will fail on the <span class="caps">CAS </span>metadata write, and then see that the ledger is in IN_RECOVERY state, and that it therefore shouldn't try to write to it.</p> + +<p>The old writer will be able to write entries to individual bookies (we can't guarantee that the fence message reaches all bookies), but as it will not be able reach ack quorum, it will not be able to send a success response to its client. The client will get a LedgerFenced error instead.</p> + +<p>It is important to note that when you get a ledger fenced message for an entry, it doesn't mean that the entry has <em>not</em> been written. It means that the entry may or may not have been written, and this can only be determined after the ledger is recovered. In effect, LedgerFenced should be treated like a timeout.</p> + +<p>Once the ledger is fenced, recovery can begin. Recovery means finding the last entry of the ledger and closing the ledger. To find the last entry of the ledger, the client asks all bookies for the highest last add confirmed value they have seen. It waits until it has received a response at least (Q<sub>w</sub>-Q<sub>a</sub>)+1 bookies from each write quorum, and takes the highest response as the entry id to start reading forward from. It then starts reading forward in the ledger, one entry at a time, replicating all entries it sees to the entire write quorum for that entry. Once it can no longer read any more entries, it updates the state in the metadata to <span class="caps">CLOSED, </span>and sets the last entry of the ledger to the last entry it wrote. Multiple readers can try to recovery a ledger at the same time, but as the metadata write is <span class="caps">CAS, </span>they will all converge on the same last entry of the ledger.</p> + +<p class="footnote" id="fn1"><sup>1</sup> Rereplication is a subsystem that runs in the background on bookies to ensure that ledgers are fully replicated even if one bookie from their ensemble is down</p> + + </div> + <footer class="footer"> + <div class="container"> + <p class="text-muted">Copyright © 2014 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.<br/> + Apache BookKeeper, BookKeeper, Apache, Apache ZooKeeper, ZooKeeper, the Apache feather logo, and the Apache BookKeeper project logo are trademarks of The Apache Software Foundation.</p> + </div> + </footer> + + <script src="http://code.jquery.com/jquery.js";></script> + <script src="/js/bootstrap.min.js"></script> + </body> +</html> Added: websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperStarted.html ============================================================================== --- websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperStarted.html (added) +++ websites/staging/bookkeeper/trunk/content/docs/r4.4.0/bookkeeperStarted.html Sun May 15 21:40:24 2016 @@ -0,0 +1,226 @@ +<!DOCTYPE html> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE- 2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> + <head> + <title>Apache BookKeeper - BookKeeper Getting Started Guide</title> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <!-- Bootstrap --> + <link href="/css/bootstrap.min.css" rel="stylesheet"> + <link href="/css/bootstrap-responsive.min.css" rel="stylesheet"> + <link href="/css/styles.css" rel="stylesheet"> + </head> + <body> + <header class="navbar navbar-inverse navbar-static-top" role="banner"> + <div class="container"> + <div class="navbar-header hidden-xs hidden-sm"> + <a class="navbar-brand navbar-logo" href="/"><img class="img-responsive" src="/img/bookkeeper_blk40.png" alt="Bookkeeper Logo" /></a> + </div> + <div class="navbar-header"> + <button class="navbar-toggle collapsed" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/">Apache BookKeeper</a> + </div> + <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> + <ul class="nav navbar-nav"> + <li><a href="/releases.html">Download</a></li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/docs/master">Latest (master)</a></li> + <li><ul> + <li><a href="/docs/master/apidocs">Java API docs</a></li> + <li><a href="/docs/master/bookkeeperTutorial.html">Tutorial</a></li> + <li><a href="/docs/master/bookkeeperConfig.html">Admin guide</a></li> + </ul><li> + <li><a href="/docs/r4.4.0">Release 4.4.0</a></li> + <li class="divider"></li> + <li>Older releases</li> + <li><a href="/docs/r4.3.2">Release 4.3.2</a></li> + <li><a href="/docs/r4.3.1">Release 4.3.1</a></li> + <li><a href="/docs/r4.3.0">Release 4.3.0</a></li> + <li><a href="/docs/r4.2.4">Release 4.2.4</a></li> + <li><a href="/docs/r4.2.3">Release 4.2.3</a></li> + <li><a href="/docs/r4.2.2">Release 4.2.2</a></li> + <li><a href="/docs/r4.2.1">Release 4.2.1</a></li> + <li><a href="/docs/r4.2.0">Release 4.2.0</a></li> + <li><a href="/docs/r4.1.0">Release 4.1.0</a></li> + <li><a href="/docs/r4.0.0">Release 4.0.0</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Get Involved<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/lists.html">Mailing Lists</a></li> + <li><a href="/irc.html">IRC</a></li> + <li><a href="/svn.html">Version Control</a></li> + <li><a href="https://issues.apache.org/jira/browse/BOOKKEEPER";>Issue Tracker</a></li> + </ul> + </li> + + <li><a href="https://cwiki.apache.org/confluence/display/BOOKKEEPER/Index";>Wiki</a></li> + <!--<li><a href="#">Hedwig</a></li>//--> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project Info<span class="caret"></span></a> + <ul class="dropdown-menu" role="menu"> + <li><a href="/credits.html">Who are we?</a></li> + <li><a href="/bylaws.html">Bylaws</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li class="divider"></li> + <li><a href="/privacy.html">Privacy Policy</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsership</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + <script> + (function() { + var cx = '017580107654524087317:iqnsyimpydg'; + var gcse = document.createElement('script'); + gcse.type = 'text/javascript'; + gcse.async = true; + gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + + '//www.google.com/cse/cse.js?cx=' + cx; + var s = document.getElementsByTagName('script')[0]; + s.parentNode.insertBefore(gcse, s); + })(); + </script> + + <div class="navbar-form navbar-right visible-lg" id="googlebox"> + <gcse:searchbox-only></gcse:searchbox-only> + </div> + </nav> + </div> + </header> + <div class="container"> + + <h1>Abstract</h1> + +<p>This guide contains detailed information about using BookKeeper for logging. It discusses the basic operations BookKeeper supports, and how to create logs and perform basic read and write operations on these logs.</p> + +<h1>Getting Started: Setting up BookKeeper to write logs.</h1> + +<p>This document contains information to get you started quickly with BookKeeper. It is aimed primarily at developers willing to try it out, and contains simple installation instructions for a simple BookKeeper installation and a simple programming example. For further programming detail, please refer to the <a href="./bookkeeperTutorial.html">BookKeeper Tutorial</a>.</p> + +<h1>Pre-requisites</h1> + +<p>See <a href="./bookkeeperConfig.html#bk_sysReqin">System Requirements</a> the Admin guide.</p> + +<h1>Download</h1> + +<p>BookKeeper trunk can be downloaded from subversion. See <a href="http://bookkeeper.apache.org/svn.html";>Version Control</a>.</p> + +<h1>LocalBookKeeper</h1> + +<p>BookKeeper provides a utility program to start a standalone ZooKeeper ensemble and a number of bookies on a local machine. As this all runs on a local machine, throughput will be very low. It should only be used for testing.</p> + +<p>To start a local bookkeeper ensemble with 5 bookies:</p> + +<p> <code>bookkeeper-server/bin/bookkeeper localbookie 5</code></p> + +<h1>Setting up bookies</h1> + +<p>If you're bold and you want more than just running things locally, then you'll need to run bookies in different servers. You'll need at least three bookies to start with. </p> + +<p>For each bookie, we need to execute a command like the following: </p> + +<p> <code>bookkeeper-server/bin/bookkeeper bookie</code></p> + +<p>This command will use the default directories for storing ledgers and the write ahead log, and will look for a zookeeper server on localhost:2181. See the <a href="./bookkeeperConfig.html">Admin Guide</a> for more details.</p> + +<p>To see the default values of these configuration variables, run:</p> + +<p> <code>bookkeeper-server/bin/bookkeeper help</code></p> + +<h1>Setting up ZooKeeper</h1> + +<p>ZooKeeper stores metadata on behalf of BookKeeper clients and bookies. To get a minimal ZooKeeper installation to work with BookKeeper, we can set up one server running in standalone mode. Once we have the server running, we need to create a few znodes: </p> + +<ol> +<li> <code>/ledgers </code> </li> +<li> <code>/ledgers/available </code> </li> +</ol> + +<p>We provide a way of bootstrapping it automatically. See the <a href="./bookkeeperConfig.html">Admin Guide</a> for a description of how to bootstrap automatically, and in particular the shell metaformat command.<br /> + </p> + +<h1>Example</h1> + +<p>In the following excerpt of code, we: </p> + +<ol> +<li>Open a bookkeeper client;</li> +<li>Create a ledger; </li> +<li>Write to the ledger; </li> +<li>Close the ledger; </li> +<li>Open the same ledger for reading; </li> +<li>Read from the ledger; </li> +<li>Close the ledger again; </li> +<li>Close the bookkeeper client.</li> +</ol> + + + +<pre><code> +BookKeeper bkc = new BookKeeper("localhost:2181"); +LedgerHandle lh = bkc.createLedger(ledgerPassword); +ledgerId = lh.getId(); +ByteBuffer entry = ByteBuffer.allocate(4); + +for(int i = 0; i < 10; i++){ + entry.putInt(i); + entry.position(0); + entries.add(entry.array()); + lh.addEntry(entry.array()); +} +lh.close(); +lh = bkc.openLedger(ledgerId, ledgerPassword); + +Enumeration<LedgerEntry> ls = lh.readEntries(0, 9); +int i = 0; +while(ls.hasMoreElements()){ + ByteBuffer origbb = ByteBuffer.wrap( + entries.get(i++)); + Integer origEntry = origbb.getInt(); + ByteBuffer result = ByteBuffer.wrap( + ls.nextElement().getEntry()); + + Integer retrEntry = result.getInt(); +} +lh.close(); +bkc.close(); +</code></pre> + + </div> + <footer class="footer"> + <div class="container"> + <p class="text-muted">Copyright © 2014 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.<br/> + Apache BookKeeper, BookKeeper, Apache, Apache ZooKeeper, ZooKeeper, the Apache feather logo, and the Apache BookKeeper project logo are trademarks of The Apache Software Foundation.</p> + </div> + </footer> + + <script src="http://code.jquery.com/jquery.js";></script> + <script src="/js/bootstrap.min.js"></script> + </body> +</html>
