http://git-wip-us.apache.org/repos/asf/zookeeper/blob/c1efa954/zookeeper-docs/src/documentation/conf/cli.xconf ---------------------------------------------------------------------- diff --git a/zookeeper-docs/src/documentation/conf/cli.xconf b/zookeeper-docs/src/documentation/conf/cli.xconf new file mode 100644 index 0000000..c671340 --- /dev/null +++ b/zookeeper-docs/src/documentation/conf/cli.xconf @@ -0,0 +1,328 @@ +<?xml version="1.0"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<!--+ + | This is the Apache Cocoon command line configuration file. + | Here you give the command line interface details of where + | to find various aspects of your Cocoon installation. + | + | If you wish, you can also use this file to specify the URIs + | that you wish to generate. + | + | The current configuration information in this file is for + | building the Cocoon documentation. Therefore, all links here + | are relative to the build context dir, which, in the build.xml + | file, is set to ${build.context} + | + | Options: + | verbose: increase amount of information presented + | to standard output (default: false) + | follow-links: whether linked pages should also be + | generated (default: true) + | precompile-only: precompile sitemaps and XSP pages, but + | do not generate any pages (default: false) + | confirm-extensions: check the mime type for the generated page + | and adjust filename and links extensions + | to match the mime type + | (e.g. text/html->.html) + | + | Note: Whilst using an xconf file to configure the Cocoon + | Command Line gives access to more features, the use of + | command line parameters is more stable, as there are + | currently plans to improve the xconf format to allow + | greater flexibility. If you require a stable and + | consistent method for accessing the CLI, it is recommended + | that you use the command line parameters to configure + | the CLI. See documentation at: + | http://cocoon.apache.org/2.1/userdocs/offline/ + | http://wiki.apache.org/cocoon/CommandLine + | + +--> + +<cocoon verbose="true" + follow-links="true" + precompile-only="false" + confirm-extensions="false"> + + <!--+ + | The context directory is usually the webapp directory + | containing the sitemap.xmap file. + | + | The config file is the cocoon.xconf file. + | + | The work directory is used by Cocoon to store temporary + | files and cache files. + | + | The destination directory is where generated pages will + | be written (assuming the 'simple' mapper is used, see + | below) + +--> + <context-dir>.</context-dir> + <config-file>WEB-INF/cocoon.xconf</config-file> + <work-dir>../tmp/cocoon-work</work-dir> + <dest-dir>../site</dest-dir> + + <!--+ + | A checksum file can be used to store checksums for pages + | as they are generated. When the site is next generated, + | files will not be written if their checksum has not changed. + | This means that it will be easier to detect which files + | need to be uploaded to a server, using the timestamp. + | + | The default path is relative to the core webapp directory. + | An asolute path can be used. + +--> + <!-- <checksums-uri>build/work/checksums</checksums-uri>--> + + <!--+ + | Broken link reporting options: + | Report into a text file, one link per line: + | <broken-links type="text" report="filename"/> + | Report into an XML file: + | <broken-links type="xml" report="filename"/> + | Ignore broken links (default): + | <broken-links type="none"/> + | + | Two attributes to this node specify whether a page should + | be generated when an error has occurred. 'generate' specifies + | whether a page should be generated (default: true) and + | extension specifies an extension that should be appended + | to the generated page's filename (default: none) + | + | Using this, a quick scan through the destination directory + | will show broken links, by their filename extension. + +--> + <broken-links type="xml" + file="../brokenlinks.xml" + generate="false" + extension=".error" + show-referrers="true"/> + + <!--+ + | Load classes at startup. This is necessary for generating + | from sites that use SQL databases and JDBC. + | The <load-class> element can be repeated if multiple classes + | are needed. + +--> + <!-- + <load-class>org.firebirdsql.jdbc.Driver</load-class> + --> + + <!--+ + | Configures logging. + | The 'log-kit' parameter specifies the location of the log kit + | configuration file (usually called logkit.xconf. + | + | Logger specifies the logging category (for all logging prior + | to other Cocoon logging categories taking over) + | + | Available log levels are: + | DEBUG: prints all level of log messages. + | INFO: prints all level of log messages except DEBUG + | ones. + | WARN: prints all level of log messages except DEBUG + | and INFO ones. + | ERROR: prints all level of log messages except DEBUG, + | INFO and WARN ones. + | FATAL_ERROR: prints only log messages of this level + +--> + <!-- <logging log-kit="WEB-INF/logkit.xconf" logger="cli" level="ERROR" /> --> + + <!--+ + | Specifies the filename to be appended to URIs that + | refer to a directory (i.e. end with a forward slash). + +--> + <default-filename>index.html</default-filename> + + <!--+ + | Specifies a user agent string to the sitemap when + | generating the site. + | + | A generic term for a web browser is "user agent". Any + | user agent, when connecting to a web server, will provide + | a string to identify itself (e.g. as Internet Explorer or + | Mozilla). It is possible to have Cocoon serve different + | content depending upon the user agent string provided by + | the browser. If your site does this, then you may want to + | use this <user-agent> entry to provide a 'fake' user agent + | to Cocoon, so that it generates the correct version of your + | site. + | + | For most sites, this can be ignored. + +--> + <!-- + <user-agent>Cocoon Command Line Environment 2.1</user-agent> + --> + + <!--+ + | Specifies an accept string to the sitemap when generating + | the site. + | User agents can specify to an HTTP server what types of content + | (by mime-type) they are able to receive. E.g. a browser may be + | able to handle jpegs, but not pngs. The HTTP accept header + | allows the server to take the browser's capabilities into account, + | and only send back content that it can handle. + | + | For most sites, this can be ignored. + +--> + + <accept>*/*</accept> + + <!--+ + | Specifies which URIs should be included or excluded, according + | to wildcard patterns. + | + | These includes/excludes are only relevant when you are following + | links. A link URI must match an include pattern (if one is given) + | and not match an exclude pattern, if it is to be followed by + | Cocoon. It can be useful, for example, where there are links in + | your site to pages that are not generated by Cocoon, such as + | references to api-documentation. + | + | By default, all URIs are included. If both include and exclude + | patterns are specified, a URI is first checked against the + | include patterns, and then against the exclude patterns. + | + | Multiple patterns can be given, using muliple include or exclude + | nodes. + | + | The order of the elements is not significant, as only the first + | successful match of each category is used. + | + | Currently, only the complete source URI can be matched (including + | any URI prefix). Future plans include destination URI matching + | and regexp matching. If you have requirements for these, contact + | [email protected]. + +--> + + <exclude pattern="**/"/> + <exclude pattern="**apidocs**"/> + <exclude pattern="api/**"/> + + <!-- ZOOKEEPER-2364 - we build our own release notes separately --> + <exclude pattern="releasenotes.**"/> + +<!-- + This is a workaround for FOR-284 "link rewriting broken when + linking to xml source views which contain site: links". + See the explanation there and in declare-broken-site-links.xsl +--> + <exclude pattern="site:**"/> + <exclude pattern="ext:**"/> + <exclude pattern="lm:**"/> + <exclude pattern="**/site:**"/> + <exclude pattern="**/ext:**"/> + <exclude pattern="**/lm:**"/> + + <!-- Exclude tokens used in URLs to ASF mirrors (interpreted by a CGI) --> + <exclude pattern="[preferred]/**"/> + <exclude pattern="[location]"/> + + <!-- <include-links extension=".html"/>--> + + <!--+ + | <uri> nodes specify the URIs that should be generated, and + | where required, what should be done with the generated pages. + | They describe the way the URI of the generated file is created + | from the source page's URI. There are three ways that a generated + | file URI can be created: append, replace and insert. + | + | The "type" attribute specifies one of (append|replace|insert): + | + | append: + | Append the generated page's URI to the end of the source URI: + | + | <uri type="append" src-prefix="documents/" src="index.html" + | dest="build/dest/"/> + | + | This means that + | (1) the "documents/index.html" page is generated + | (2) the file will be written to "build/dest/documents/index.html" + | + | replace: + | Completely ignore the generated page's URI - just + | use the destination URI: + | + | <uri type="replace" src-prefix="documents/" src="index.html" + | dest="build/dest/docs.html"/> + | + | This means that + | (1) the "documents/index.html" page is generated + | (2) the result is written to "build/dest/docs.html" + | (3) this works only for "single" pages - and not when links + | are followed + | + | insert: + | Insert generated page's URI into the destination + | URI at the point marked with a * (example uses fictional + | zip protocol) + | + | <uri type="insert" src-prefix="documents/" src="index.html" + | dest="zip://*.zip/page.html"/> + | + | This means that + | (1) + | + | In any of these scenarios, if the dest attribute is omitted, + | the value provided globally using the <dest-dir> node will + | be used instead. + +--> + <!-- + <uri type="replace" + src-prefix="samples/" + src="hello-world/hello.html" + dest="build/dest/hello-world.html"/> + --> + + <!--+ + | <uri> nodes can be grouped together in a <uris> node. This + | enables a group of URIs to share properties. The following + | properties can be set for a group of URIs: + | * follow-links: should pages be crawled for links + | * confirm-extensions: should file extensions be checked + | for the correct mime type + | * src-prefix: all source URIs should be + | pre-pended with this prefix before + | generation. The prefix is not + | included when calculating the + | destination URI + | * dest: the base destination URI to be + | shared by all pages in this group + | * type: the method to be used to calculate + | the destination URI. See above + | section on <uri> node for details. + | + | Each <uris> node can have a name attribute. When a name + | attribute has been specified, the -n switch on the command + | line can be used to tell Cocoon to only process the URIs + | within this URI group. When no -n switch is given, all + | <uris> nodes are processed. Thus, one xconf file can be + | used to manage multiple sites. + +--> + <!-- + <uris name="mirrors" follow-links="false"> + <uri type="append" src="mirrors.html"/> + </uris> + --> + + <!--+ + | File containing URIs (plain text, one per line). + +--> + <!-- + <uri-file>uris.txt</uri-file> + --> +</cocoon>
http://git-wip-us.apache.org/repos/asf/zookeeper/blob/c1efa954/zookeeper-docs/src/documentation/content/xdocs/bookkeeperConfig.xml ---------------------------------------------------------------------- diff --git a/zookeeper-docs/src/documentation/content/xdocs/bookkeeperConfig.xml b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperConfig.xml new file mode 100644 index 0000000..7a80949 --- /dev/null +++ b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperConfig.xml @@ -0,0 +1,156 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Copyright 2002-2004 The Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN" +"http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd";> +<article id="bk_Admin"> + <title>BookKeeper Administrator's Guide</title> + + <subtitle>Setup Guide</subtitle> + + <articleinfo> + <legalnotice> + <para>Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. You may + obtain a copy of the License at <ulink + url="http://www.apache.org/licenses/LICENSE-2.0";>http://www.apache.org/licenses/LICENSE-2.0</ulink>. + </para> + + <para>Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an "AS IS" + BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing permissions + and limitations under the License. + </para> + </legalnotice> + + <abstract> + <para>This document contains information about deploying, administering + and mantaining BookKeeper. It also discusses best practices and common + problems. + </para> + <para> As BookKeeper is still a prototype, this article is likely to change + significantly over time. + </para> + </abstract> + </articleinfo> + + <section id="bk_deployment"> + <title>Deployment</title> + + <para>This section contains information about deploying BookKeeper and + covers these topics:</para> + + <itemizedlist> + <listitem> + <para><xref linkend="bk_sysReq" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_runningBookies" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_zkMetadata" /></para> + </listitem> + </itemizedlist> + + <para> The first section tells you how many machines you need. The second explains how to bootstrap bookies + (BookKeeper storage servers). The third section explains how we use ZooKeeper and our requirements with + respect to ZooKeeper. + </para> + + <section id="bk_sysReq"> + <title>System requirements</title> + <para> A typical BookKeeper installation comprises a set of bookies and a set of ZooKeeper replicas. The exact number of bookies + depends on the quorum mode, desired throughput, and number of clients using this installation simultaneously. The minimum number of + bookies is three for self-verifying (stores a message authentication code along with each entry) and four for generic (does not + store a message authentication codewith each entry), and there is no upper limit on the number of bookies. Increasing the number of + bookies, in fact, enables higher throughput. + </para> + + <para> For performance, we require each server to have at least two disks. It is possible to run a bookie with a single disk, but + performance will be significantly lower in this case. Of course, it works with one disk, but performance is significantly lower. + </para> + + <para> For ZooKeeper, there is no constraint with respect to the number of replicas. Having a single machine running ZooKeeper + in standalone mode is sufficient for BookKeeper. For resilience purposes, it might be a good idea to run ZooKeeper in quorum + mode with multiple servers. Please refer to the ZooKeeper documentation for detail on how to configure ZooKeeper with multiple + replicas + </para> + </section> + + <section id="bk_runningBookies"> + <title>Running bookies</title> + <para> + To run a bookie, we execute the following command: + </para> + + <para><computeroutput> + java -cp .:./zookeeper-<version>-bookkeeper.jar:./zookeeper-<version>.jar\ + :../log4j/apache-log4j-1.2.15/log4j-1.2.15.jar -Dlog4j.configuration=log4j.properties\ + org.apache.bookkeeper.proto.BookieServer 3181 127.0.0.1:2181 /path_to_log_device/\ + /path_to_ledger_device/ + </computeroutput></para> + + <para> + The parameters are: + </para> + + <itemizedlist> + <listitem> + <para> + Port number that the bookie listens on; + </para> + </listitem> + + <listitem> + <para> + Comma separated list of ZooKeeper servers with a hostname:port format; + </para> + </listitem> + + <listitem> + <para> + Path for Log Device (stores bookie write-ahead log); + </para> + </listitem> + + <listitem> + <para> + Path for Ledger Device (stores ledger entries); + </para> + </listitem> + </itemizedlist> + + <para> + Ideally, <computeroutput>/path_to_log_device/ </computeroutput> and <computeroutput>/path_to_ledger_device/ </computeroutput> are each + in a different device. + </para> + </section> + + <section id="bk_zkMetadata"> + <title>ZooKeeper Metadata</title> + <para> + For BookKeeper, we require a ZooKeeper installation to store metadata, and to pass the list + of ZooKeeper servers as parameter to the constructor of the BookKeeper class (<computeroutput> + org.apache.bookkeeper.client,BookKeeper</computeroutput>). + To setup ZooKeeper, please check the <ulink url="index.html"> + ZooKeeper documentation</ulink>. + </para> + </section> + </section> +</article> http://git-wip-us.apache.org/repos/asf/zookeeper/blob/c1efa954/zookeeper-docs/src/documentation/content/xdocs/bookkeeperOverview.xml ---------------------------------------------------------------------- diff --git a/zookeeper-docs/src/documentation/content/xdocs/bookkeeperOverview.xml b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperOverview.xml new file mode 100644 index 0000000..cdc1878 --- /dev/null +++ b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperOverview.xml @@ -0,0 +1,419 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Copyright 2002-2004 The Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN" +"http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd";> +<article id="bk_GettStartedGuide"> + <title>BookKeeper overview</title> + + <articleinfo> + <legalnotice> + <para>Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. You may + obtain a copy of the License at <ulink + url="http://www.apache.org/licenses/LICENSE-2.0";>http://www.apache.org/licenses/LICENSE-2.0</ulink>.</para> + + <para>Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an "AS IS" + BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing permissions + and limitations under the License.</para> + </legalnotice> + + <abstract> + <para>This guide contains detailed information about using BookKeeper + for logging. It discusses the basic operations BookKeeper supports, + and how to create logs and perform basic read and write operations on these + logs.</para> + </abstract> + </articleinfo> + <section id="bk_Overview"> + <title>BookKeeper overview</title> + + <section id="bk_Intro"> + <title>BookKeeper introduction</title> + <para> + BookKeeper is a replicated service to reliably log streams of records. In BookKeeper, + servers are "bookies", log streams are "ledgers", and each unit of a log (aka record) is a + "ledger entry". BookKeeper is designed to be reliable; bookies, the servers that store + ledgers, can crash, corrupt data, discard data, but as long as there are enough bookies + behaving correctly the service as a whole behaves correctly. + </para> + + <para> + The initial motivation for BookKeeper comes from the namenode of HDFS. Namenodes have to + log operations in a reliable fashion so that recovery is possible in the case of crashes. + We have found the applications for BookKeeper extend far beyond HDFS, however. Essentially, + any application that requires an append storage can replace their implementations with + BookKeeper. BookKeeper has the advantage of scaling throughput with the number of servers. + </para> + + <para> + At a high level, a bookkeeper client receives entries from a client application and stores it to + sets of bookies, and there are a few advantages in having such a service: + </para> + + <itemizedlist> + <listitem> + <para> + We can use hardware that is optimized for such a service. We currently believe that such a + system has to be optimized only for disk I/O; + </para> + </listitem> + + <listitem> + <para> + We can have a pool of servers implementing such a log system, and shared among a number of servers; + </para> + </listitem> + + <listitem> + <para> + We can have a higher degree of replication with such a pool, which makes sense if the hardware necessary for it is cheaper compared to the one the application uses. + </para> + </listitem> + </itemizedlist> + + </section> + + <section id="bk_moreDetail"> + <title>In slightly more detail...</title> + + <para> BookKeeper implements highly available logs, and it has been designed with write-ahead logging in mind. Besides high availability + due to the replicated nature of the service, it provides high throughput due to striping. As we write entries in a subset of bookies of an + ensemble and rotate writes across available quorums, we are able to increase throughput with the number of servers for both reads and writes. + Scalability is a property that is possible to achieve in this case due to the use of quorums. Other replication techniques, such as + state-machine replication, do not enable such a property. + </para> + + <para> An application first creates a ledger before writing to bookies through a local BookKeeper client instance. + Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently + has a single writer. This writer has to execute a close ledger operation before any other client can read from it. + If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the + opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover + it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery + procedure simply finds the last entry written correctly and writes it to ZooKeeper. + </para> + + <para> + Note that currently this recovery procedure is executed automatically upon trying to open a ledger and no explicit action is necessary. + Although two clients may try to recover a ledger concurrently, only one will succeed, the first one that is able to create the close znode + for the ledger. + </para> + </section> + + <section id="bk_basicComponents"> + <title>Bookkeeper elements and concepts</title> + <para> + BookKeeper uses four basic elements: + </para> + + <itemizedlist> + <listitem> + <para> + <emphasis role="bold">Ledger</emphasis>: A ledger is a sequence of entries, and each entry is a sequence of bytes. Entries are + written sequentially to a ledger and at most once. Consequently, ledgers have an append-only semantics; + </para> + </listitem> + + <listitem> + <para> + <emphasis role="bold">BookKeeper client</emphasis>: A client runs along with a BookKeeper application, and it enables applications + to execute operations on ledgers, such as creating a ledger and writing to it; + </para> + </listitem> + + <listitem> + <para> + <emphasis role="bold">Bookie</emphasis>: A bookie is a BookKeeper storage server. Bookies store the content of ledgers. For any given + ledger L, we call an <emphasis>ensemble</emphasis> the group of bookies storing the content of L. For performance, we store on + each bookie of an ensemble only a fragment of a ledger. That is, we stripe when writing entries to a ledger such that + each entry is written to sub-group of bookies of the ensemble. + </para> + </listitem> + + <listitem> + <para> + <emphasis role="bold">Metadata storage service</emphasis>: BookKeeper requires a metadata storage service to store information related + to ledgers and available bookies. We currently use ZooKeeper for such a task. + </para> + </listitem> + </itemizedlist> + </section> + + <section id="bk_initialDesign"> + <title>Bookkeeper initial design</title> + <para> + A set of bookies implements BookKeeper, and we use a quorum-based protocol to replicate data across the bookies. + There are basically two operations to an existing ledger: read and append. Here is the complete API list + (mode detail <ulink url="bookkeeperProgrammer.html"> + here</ulink>): + </para> + + <itemizedlist> + <listitem> + <para> + Create ledger: creates a new empty ledger; + </para> + </listitem> + + <listitem> + <para> + Open ledger: opens an existing ledger for reading; + </para> + </listitem> + + <listitem> + <para> + Add entry: adds a record to a ledger either synchronously or asynchronously; + </para> + </listitem> + + <listitem> + <para> + Read entries: reads a sequence of entries from a ledger either synchronously or asynchronously + </para> + </listitem> + </itemizedlist> + + <para> + There is only a single client that can write to a ledger. Once that ledger is closed or the client fails, + no more entries can be added. (We take advantage of this behavior to provide our strong guarantees.) + There will not be gaps in the ledger. Fingers get broken, people get roughed up or end up in prison when + books are manipulated, so there is no deleting or changing of entries. + </para> + + <figure> + <title>BookKeeper Overview</title> + + <mediaobject> + <imageobject> + <imagedata fileref="images/bk-overview.jpg" width="3in" depth="3in" contentwidth="3in" contentdepth="3in" scalefit="0"/> + </imageobject> + </mediaobject> + </figure> + + <para> + A simple use of BooKeeper is to implement a write-ahead transaction log. A server maintains an in-memory data structure + (with periodic snapshots for example) and logs changes to that structure before it applies the change. The application + server creates a ledger at startup and store the ledger id and password in a well known place (ZooKeeper maybe). When + it needs to make a change, the server adds an entry with the change information to a ledger and apply the change when + BookKeeper adds the entry successfully. The server can even use asyncAddEntry to queue up many changes for high change + throughput. BooKeeper meticulously logs the changes in order and call the completion functions in order. + </para> + + <para> + When the application server dies, a backup server will come online, get the last snapshot and then it will open the + ledger of the old server and read all the entries from the time the snapshot was taken. (Since it doesn't know the + last entry number it will use MAX_INTEGER). Once all the entries have been processed, it will close the ledger and + start a new one for its use. + </para> + + <para> + A client library takes care of communicating with bookies and managing entry numbers. An entry has the following fields: + </para> + + <table frame='all'><title>Entry fields</title> + <tgroup cols='3' align='left' colsep='1' rowsep='1'> + <colspec colname='Field'/> + <colspec colname='Type'/> + <colspec colname='Description'/> + <colspec colnum='5' colname='c5'/> + <thead> + <row> + <entry>Field</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + <tfoot> + <row> + <entry>Ledger number</entry> + <entry>long</entry> + <entry>The id of the ledger of this entry</entry> + </row> + <row> + <entry>Entry number</entry> + <entry>long</entry> + <entry>The id of this entry</entry> + </row> + </tfoot> + <tbody> + <row> + <entry>last confirmed (<emphasis>LC</emphasis>)</entry> + <entry>long</entry> + <entry>id of the last recorded entry</entry> + </row> + <row> + <entry>data</entry> + <entry>byte[]</entry> + <entry>the entry data (supplied by application)</entry> + </row> + <row> + <entry>authentication code</entry> + <entry>byte[]</entry> + <entry>Message authentication code that includes all other fields of the entry</entry> + </row> + + </tbody> + </tgroup> + </table> + + <para> + The client library generates a ledger entry. None of the fields are modified by the bookies and only the first three + fields are interpreted by the bookies. + </para> + + <para> + To add to a ledger, the client generates the entry above using the ledger number. The entry number will be one more + than the last entry generated. The <emphasis>LC</emphasis> field contains the last entry that has been successfully recorded by BookKeeper. + If the client writes entries one at a time, <emphasis>LC</emphasis> is the last entry id. But, if the client is using asyncAddEntry, there + may be many entries in flight. An entry is considered recorded when both of the following conditions are met: + </para> + + <itemizedlist> + <listitem> + <para> + the entry has been accepted by a quorum of bookies + </para> + </listitem> + + <listitem> + <para> + all entries with a lower entry id have been accepted by a quorum of bookies + </para> + </listitem> + </itemizedlist> + + <para> + <emphasis>LC</emphasis> seems mysterious right now, but it is too early to explain how we use it; just smile and move on. + </para> + + <para> + Once all the other fields have been field in, the client generates an authentication code with all of the previous fields. + The entry is then sent to a quorum of bookies to be recorded. Any failures will result in the entry being sent to a new + quorum of bookies. + </para> + + <para> + To read, the client library initially contacts a bookie and starts requesting entries. If an entry is missing or + invalid (a bad MAC for example), the client will make a request to a different bookie. By using quorum writes, + as long as enough bookies are up we are guaranteed to eventually be able to read an entry. + </para> + + </section> + + <section id="bk_metadata"> + <title>Bookkeeper metadata management</title> + + <para> + There are some meta data that needs to be made available to BookKeeper clients: + </para> + + <itemizedlist> + <listitem> + <para> + The available bookies; + </para> + </listitem> + + <listitem> + <para> + The list of ledgers; + </para> + </listitem> + + <listitem> + <para> + The list of bookies that have been used for a given ledger; + </para> + </listitem> + + <listitem> + <para> + The last entry of a ledger; + </para> + </listitem> + </itemizedlist> + + <para> + We maintain this information in ZooKeeper. Bookies use ephemeral nodes to indicate their availability. Clients + use znodes to track ledger creation and deletion and also to know the end of the ledger and the bookies that + were used to store the ledger. Bookies also watch the ledger list so that they can cleanup ledgers that get deleted. + </para> + + </section> + + <section id="bk_closingOut"> + <title>Closing out ledgers</title> + + <para> + The process of closing out the ledger and finding the last ledger is difficult due to the durability guarantees of BookKeeper: + </para> + + <itemizedlist> + <listitem> + <para> + If an entry has been successfully recorded, it must be readable. + </para> + </listitem> + + <listitem> + <para> + If an entry is read once, it must always be available to be read. + </para> + </listitem> + </itemizedlist> + + <para> + If the ledger was closed gracefully, ZooKeeper will have the last entry and everything will work well. But, if the + BookKeeper client that was writing the ledger dies, there is some recovery that needs to take place. + </para> + + <para> + The problematic entries are the ones at the end of the ledger. There can be entries in flight when a BookKeeper client + dies. If the entry only gets to one bookie, the entry should not be readable since the entry will disappear if that bookie + fails. If the entry is only on one bookie, that doesn't mean that the entry has not been recorded successfully; the other + bookies that recorded the entry might have failed. + </para> + + <para> + The trick to making everything work is to have a correct idea of a last entry. We do it in roughly three steps: + </para> + <orderedlist> + <listitem> + <para> + Find the entry with the highest last recorded entry, <emphasis>LC</emphasis>; + </para> + </listitem> + + <listitem> + <para> + Find the highest consecutively recorded entry, <emphasis>LR</emphasis>; + </para> + </listitem> + + <listitem> + <para> + Make sure that all entries between <emphasis>LC</emphasis> and <emphasis>LR</emphasis> are on a quorum of bookies; + </para> + </listitem> + + </orderedlist> + </section> + </section> +</article> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/zookeeper/blob/c1efa954/zookeeper-docs/src/documentation/content/xdocs/bookkeeperProgrammer.xml ---------------------------------------------------------------------- diff --git a/zookeeper-docs/src/documentation/content/xdocs/bookkeeperProgrammer.xml b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperProgrammer.xml new file mode 100644 index 0000000..5f330e1 --- /dev/null +++ b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperProgrammer.xml @@ -0,0 +1,678 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Copyright 2002-2004 The Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN" +"http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd";> +<article id="bk_GettStartedGuide"> + <title>BookKeeper Getting Started Guide</title> + + <articleinfo> + <legalnotice> + <para>Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. You may + obtain a copy of the License at <ulink + url="http://www.apache.org/licenses/LICENSE-2.0";>http://www.apache.org/licenses/LICENSE-2.0</ulink>.</para> + + <para>Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an "AS IS" + BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing permissions + and limitations under the License.</para> + </legalnotice> + + <abstract> + <para>This guide contains detailed information about using BookKeeper + for logging. It discusses the basic operations BookKeeper supports, + and how to create logs and perform basic read and write operations on these + logs.</para> + </abstract> + </articleinfo> + <section id="bk_GettingStarted"> + <title>Programming with BookKeeper</title> + + <itemizedlist> + <listitem> + <para><xref linkend="bk_instance" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_createLedger" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_writeLedger" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_closeLedger" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_openLedger" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_readLedger" /></para> + </listitem> + + <listitem> + <para><xref linkend="bk_deleteLedger" /></para> + </listitem> + + </itemizedlist> + + <section id="bk_instance"> + <title> Instantiating BookKeeper.</title> + <para> + The first step to use BookKeeper is to instantiate a BookKeeper object: + </para> + <para> + <computeroutput> + org.apache.bookkeeper.BookKeeper + </computeroutput> + </para> + + <para> + There are three BookKeeper constructors: + </para> + + <para> + <computeroutput> + public BookKeeper(String servers) + throws KeeperException, IOException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>servers</computeroutput> is a comma-separated list of ZooKeeper servers. + </para> + </listitem> + </itemizedlist> + + <para> + <computeroutput> + public BookKeeper(ZooKeeper zk) + throws InterruptedException, KeeperException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>zk</computeroutput> is a ZooKeeper object. This constructor is useful when + the application also using ZooKeeper and wants to have a single instance of ZooKeeper. + </para> + </listitem> + </itemizedlist> + + + <para> + <computeroutput> + public BookKeeper(ZooKeeper zk, ClientSocketChannelFactory channelFactory) + throws InterruptedException, KeeperException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>zk</computeroutput> is a ZooKeeper object. This constructor is useful when + the application also using ZooKeeper and wants to have a single instance of ZooKeeper. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>channelFactory</computeroutput> is a netty channel object + (<computeroutput>org.jboss.netty.channel.socket</computeroutput>). + </para> + </listitem> + </itemizedlist> + + + + </section> + + <section id="bk_createLedger"> + <title> Creating a ledger. </title> + + <para> Before writing entries to BookKeeper, it is necessary to create a ledger. + With the current BookKeeper API, it is possible to create a ledger both synchronously + or asynchronously. The following methods belong + to <computeroutput>org.apache.bookkeeper.client.BookKeeper</computeroutput>. + </para> + + <para> + <emphasis role="bold">Synchronous call:</emphasis> + </para> + + <para> + <computeroutput> + public LedgerHandle createLedger(int ensSize, int qSize, DigestType type, byte passwd[]) + throws KeeperException, InterruptedException, + IOException, BKException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>ensSize</computeroutput> is the number of bookies (ensemble size); + </para> + </listitem> + + <listitem> + <para> + <computeroutput>qSize</computeroutput> is the write quorum size; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>type</computeroutput> is the type of digest used with entries: either MAC or CRC32. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>passwd</computeroutput> is a password that authorizes the client to write to the + ledger being created. + </para> + </listitem> + </itemizedlist> + + <para> + All further operations on a ledger are invoked through the <computeroutput>LedgerHandle</computeroutput> + object returned. + </para> + + <para> + As a convenience, we provide a <computeroutput>createLedger</computeroutput> with default parameters (3,2,VERIFIABLE), + and the only two input parameters it requires are a digest type and a password. + </para> + + <para> + <emphasis role="bold">Asynchronous call:</emphasis> + </para> + + <para> + <computeroutput> + public void asyncCreateLedger(int ensSize, + int qSize, + DigestType type, + byte passwd[], + CreateCallback cb, + Object ctx + ) + </computeroutput> + </para> + + <para> + The parameters are the same of the synchronous version, with the + exception of <computeroutput>cb</computeroutput> and <computeroutput>ctx</computeroutput>. <computeroutput>CreateCallback</computeroutput> + is an interface in <computeroutput>org.apache.bookkeeper.client.AsyncCallback</computeroutput>, and + a class implementing it has to implement a method called <computeroutput>createComplete</computeroutput> + that has the following signature: + </para> + + <para> + <computeroutput> + void createComplete(int rc, LedgerHandle lh, Object ctx); + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>rc</computeroutput> is a return code (please refer to <computeroutput>org.apache.bookeeper.client.BKException</computeroutput> for a list); + </para> + </listitem> + + <listitem> + <para> + <computeroutput>lh</computeroutput> is a <computeroutput>LedgerHandle</computeroutput> object to manipulate a ledger; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>ctx</computeroutput> is a control object for accountability purposes. It can be essentially any object the application is happy with. + </para> + </listitem> + </itemizedlist> + + <para> + The <computeroutput>ctx</computeroutput> object passed as a parameter to the call to create a ledger + is the one same returned in the callback. + </para> + </section> + + <section id="bk_writeLedger"> + <title> Adding entries to a ledger. </title> + <para> + Once we have a ledger handle <computeroutput>lh</computeroutput> obtained through a call to create a ledger, we + can start writing entries. As with creating ledgers, we can write both synchronously and + asynchronously. The following methods belong + to <computeroutput>org.apache.bookkeeper.client.LedgerHandle</computeroutput>. + </para> + + <para> + <emphasis role="bold">Synchronous call:</emphasis> + </para> + + <para> + <computeroutput> + public long addEntry(byte[] data) + throws InterruptedException + </computeroutput> + </para> + + <para> + where: + </para> + + <itemizedlist> + <listitem> + <para> + <computeroutput>data</computeroutput> is a byte array; + </para> + </listitem> + </itemizedlist> + + <para> + A call to <computeroutput>addEntry</computeroutput> returns the status of the operation (please refer to <computeroutput>org.apache.bookeeper.client.BKDefs</computeroutput> for a list); + </para> + + <para> + <emphasis role="bold">Asynchronous call:</emphasis> + </para> + + <para> + <computeroutput> + public void asyncAddEntry(byte[] data, AddCallback cb, Object ctx) + </computeroutput> + </para> + + <para> + It also takes a byte array as the sequence of bytes to be stored as an entry. Additionaly, it takes + a callback object <computeroutput>cb</computeroutput> and a control object <computeroutput>ctx</computeroutput>. The callback object must implement + the <computeroutput>AddCallback</computeroutput> interface in <computeroutput>org.apache.bookkeeper.client.AsyncCallback</computeroutput>, and + a class implementing it has to implement a method called <computeroutput>addComplete</computeroutput> + that has the following signature: + </para> + + <para> + <computeroutput> + void addComplete(int rc, LedgerHandle lh, long entryId, Object ctx); + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>rc</computeroutput> is a return code (please refer to <computeroutput>org.apache.bookeeper.client.BKDefs</computeroutput> for a list); + </para> + </listitem> + + <listitem> + <para> + <computeroutput>lh</computeroutput> is a <computeroutput>LedgerHandle</computeroutput> object to manipulate a ledger; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>entryId</computeroutput> is the identifier of entry associated with this request; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>ctx</computeroutput> is control object used for accountability purposes. It can be any object the application is happy with. + </para> + </listitem> + </itemizedlist> + </section> + + <section id="bk_closeLedger"> + <title> Closing a ledger. </title> + <para> + Once a client is done writing, it closes the ledger. The following methods belong + to <computeroutput>org.apache.bookkeeper.client.LedgerHandle</computeroutput>. + </para> + <para> + <emphasis role="bold">Synchronous close:</emphasis> + </para> + + <para> + <computeroutput> + public void close() + throws InterruptedException + </computeroutput> + </para> + + <para> + It takes no input parameters. + </para> + + <para> + <emphasis role="bold">Asynchronous close:</emphasis> + </para> + <para> + <computeroutput> + public void asyncClose(CloseCallback cb, Object ctx) + throws InterruptedException + </computeroutput> + </para> + + <para> + It takes a callback object <computeroutput>cb</computeroutput> and a control object <computeroutput>ctx</computeroutput>. The callback object must implement + the <computeroutput>CloseCallback</computeroutput> interface in <computeroutput>org.apache.bookkeeper.client.AsyncCallback</computeroutput>, and + a class implementing it has to implement a method called <computeroutput>closeComplete</computeroutput> + that has the following signature: + </para> + + <para> + <computeroutput> + void closeComplete(int rc, LedgerHandle lh, Object ctx) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>rc</computeroutput> is a return code (please refer to <computeroutput>org.apache.bookeeper.client.BKDefs</computeroutput> for a list); + </para> + </listitem> + + <listitem> + <para> + <computeroutput>lh</computeroutput> is a <computeroutput>LedgerHandle</computeroutput> object to manipulate a ledger; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>ctx</computeroutput> is control object used for accountability purposes. + </para> + </listitem> + </itemizedlist> + + </section> + + <section id="bk_openLedger"> + <title> Opening a ledger. </title> + <para> + To read from a ledger, a client must open it first. The following methods belong + to <computeroutput>org.apache.bookkeeper.client.BookKeeper</computeroutput>. + </para> + + <para> + <emphasis role="bold">Synchronous open:</emphasis> + </para> + + <para> + <computeroutput> + public LedgerHandle openLedger(long lId, DigestType type, byte passwd[]) + throws InterruptedException, BKException + </computeroutput> + </para> + + <itemizedlist> + <listitem> + <para> + <computeroutput>ledgerId</computeroutput> is the ledger identifier; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>type</computeroutput> is the type of digest used with entries: either MAC or CRC32. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>passwd</computeroutput> is a password to access the ledger (used only in the case of <computeroutput>VERIFIABLE</computeroutput> ledgers); + </para> + </listitem> + </itemizedlist> + + <para> + <emphasis role="bold">Asynchronous open:</emphasis> + </para> + <para> + <computeroutput> + public void asyncOpenLedger(long lId, DigestType type, byte passwd[], OpenCallback cb, Object ctx) + </computeroutput> + </para> + + <para> + It also takes a a ledger identifier and a password. Additionaly, it takes a callback object + <computeroutput>cb</computeroutput> and a control object <computeroutput>ctx</computeroutput>. The callback object must implement + the <computeroutput>OpenCallback</computeroutput> interface in <computeroutput>org.apache.bookkeeper.client.AsyncCallback</computeroutput>, and + a class implementing it has to implement a method called <computeroutput>openComplete</computeroutput> + that has the following signature: + </para> + + <para> + <computeroutput> + public void openComplete(int rc, LedgerHandle lh, Object ctx) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>rc</computeroutput> is a return code (please refer to <computeroutput>org.apache.bookeeper.client.BKDefs</computeroutput> for a list); + </para> + </listitem> + + <listitem> + <para> + <computeroutput>lh</computeroutput> is a <computeroutput>LedgerHandle</computeroutput> object to manipulate a ledger; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>ctx</computeroutput> is control object used for accountability purposes. + </para> + </listitem> + </itemizedlist> + </section> + + <section id="bk_readLedger"> + <title> Reading from ledger </title> + <para> + Read calls may request one or more consecutive entries. The following methods belong + to <computeroutput>org.apache.bookkeeper.client.LedgerHandle</computeroutput>. + </para> + + <para> + <emphasis role="bold">Synchronous read:</emphasis> + </para> + + <para> + <computeroutput> + public Enumeration<LedgerEntry> readEntries(long firstEntry, long lastEntry) + throws InterruptedException, BKException + </computeroutput> + </para> + + <itemizedlist> + <listitem> + <para> + <computeroutput>firstEntry</computeroutput> is the identifier of the first entry in the sequence of entries to read; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>lastEntry</computeroutput> is the identifier of the last entry in the sequence of entries to read. + </para> + </listitem> + </itemizedlist> + + <para> + <emphasis role="bold">Asynchronous read:</emphasis> + </para> + <para> + <computeroutput> + public void asyncReadEntries(long firstEntry, + long lastEntry, ReadCallback cb, Object ctx) + throws BKException, InterruptedException + </computeroutput> + </para> + + <para> + It also takes a first and a last entry identifiers. Additionaly, it takes a callback object + <computeroutput>cb</computeroutput> and a control object <computeroutput>ctx</computeroutput>. The callback object must implement + the <computeroutput>ReadCallback</computeroutput> interface in <computeroutput>org.apache.bookkeeper.client.AsyncCallback</computeroutput>, and + a class implementing it has to implement a method called <computeroutput>readComplete</computeroutput> + that has the following signature: + </para> + + <para> + <computeroutput> + void readComplete(int rc, LedgerHandle lh, Enumeration<LedgerEntry> seq, Object ctx) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>rc</computeroutput> is a return code (please refer to <computeroutput>org.apache.bookeeper.client.BKDefs</computeroutput> for a list); + </para> + </listitem> + + <listitem> + <para> + <computeroutput>lh</computeroutput> is a <computeroutput>LedgerHandle</computeroutput> object to manipulate a ledger; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>seq</computeroutput> is a <computeroutput>Enumeration<LedgerEntry> </computeroutput> object to containing the list of entries requested; + </para> + </listitem> + + <listitem> + <para> + <computeroutput>ctx</computeroutput> is control object used for accountability purposes. + </para> + </listitem> + </itemizedlist> + </section> + + <section id="bk_deleteLedger"> + <title> Deleting a ledger </title> + <para> + Once a client is done with a ledger and is sure that nobody will ever need to read from it again, they can delete the ledger. + The following methods belong to <computeroutput>org.apache.bookkeeper.client.BookKeeper</computeroutput>. + </para> + + <para> + <emphasis role="bold">Synchronous delete:</emphasis> + </para> + + <para> + <computeroutput> + public void deleteLedger(long lId) throws InterruptedException, BKException + </computeroutput> + </para> + + <itemizedlist> + <listitem> + <para> + <computeroutput>lId</computeroutput> is the ledger identifier; + </para> + </listitem> + </itemizedlist> + + <para> + <emphasis role="bold">Asynchronous delete:</emphasis> + </para> + <para> + <computeroutput> + public void asyncDeleteLedger(long lId, DeleteCallback cb, Object ctx) + </computeroutput> + </para> + + <para> + It takes a ledger identifier. Additionally, it takes a callback object + <computeroutput>cb</computeroutput> and a control object <computeroutput>ctx</computeroutput>. The callback object must implement + the <computeroutput>DeleteCallback</computeroutput> interface in <computeroutput>org.apache.bookkeeper.client.AsyncCallback</computeroutput>, and + a class implementing it has to implement a method called <computeroutput>deleteComplete</computeroutput> + that has the following signature: + </para> + + <para> + <computeroutput> + void deleteComplete(int rc, Object ctx) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>rc</computeroutput> is a return code (please refer to <computeroutput>org.apache.bookeeper.client.BKDefs</computeroutput> for a list); + </para> + </listitem> + + <listitem> + <para> + <computeroutput>ctx</computeroutput> is control object used for accountability purposes. + </para> + </listitem> + </itemizedlist> + </section> + </section> +</article> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/zookeeper/blob/c1efa954/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStarted.xml ---------------------------------------------------------------------- diff --git a/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStarted.xml b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStarted.xml new file mode 100644 index 0000000..74f6f7e --- /dev/null +++ b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStarted.xml @@ -0,0 +1,208 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Copyright 2002-2004 The Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN" +"http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd";> +<article id="bk_GettStartedGuide"> + <title>BookKeeper Getting Started Guide</title> + + <articleinfo> + <legalnotice> + <para>Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. You may + obtain a copy of the License at <ulink + url="http://www.apache.org/licenses/LICENSE-2.0";>http://www.apache.org/licenses/LICENSE-2.0</ulink>.</para> + + <para>Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an "AS IS" + BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing permissions + and limitations under the License.</para> + </legalnotice> + + <abstract> + <para>This guide contains detailed information about using BookKeeper + for logging. It discusses the basic operations BookKeeper supports, + and how to create logs and perform basic read and write operations on these + logs.</para> + </abstract> + </articleinfo> + <section id="bk_GettingStarted"> + <title>Getting Started: Setting up BookKeeper to write logs.</title> + + <para>This document contains information to get you started quickly with + BookKeeper. It is aimed primarily at developers willing to try it out, and + contains simple installation instructions for a simple BookKeeper installation + and a simple programming example. For further programming detail, please refer to + <ulink url="bookkeeperProgrammer.html">BookKeeper Programmer's Guide</ulink>. + </para> + + <section id="bk_Prerequisites"> + <title>Pre-requisites</title> + <para>See <ulink url="bookkeeperConfig.html#bk_sysReq"> + System Requirements</ulink> in the Admin guide.</para> + </section> + + <section id="bk_Download"> + <title>Download</title> + <para> BookKeeper is distributed along with ZooKeeper. To get a ZooKeeper distribution, + download a recent + <ulink url="http://zookeeper.apache.org/releases.html";> + stable</ulink> release from one of the Apache Download + Mirrors.</para> + </section> + + <section id="bk_localBK"> + <title>LocalBookKeeper</title> + <para> Under org.apache.bookkeeper.util, you'll find a java program + called LocalBookKeeper.java that sets you up to run BookKeeper on a + single machine. This is far from ideal from a performance perspective, + but the program is useful for both test and educational purposes. + </para> + </section> + + <section id="bk_setupBookies"> + <title>Setting up bookies</title> + <para> If you're bold and you want more than just running things locally, then + you'll need to run bookies in different servers. You'll need at least three bookies + to start with. + </para> + + <para> + For each bookie, we need to execute a command like the following: + </para> + + <para><computeroutput> + java -cp .:./zookeeper-<version>-bookkeeper.jar:./zookeeper-<version>.jar\ + :lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar -Dlog4j.configuration=log4j.properties\ + org.apache.bookkeeper.proto.BookieServer 3181 127.0.0.1:2181 /path_to_log_device/\ + /path_to_ledger_device/ + </computeroutput></para> + + <para> "/path_to_log_device/" and "/path_to_ledger_device/" are different paths. Also, port 3181 + is the port that a bookie listens on for connection requests from clients. 127.0.0.1:2181 is the hostname:port + for the ZooKeeper server. In this example, the standalone ZooKeeper server is running locally on port 2181. + If we had multiple ZooKeeper servers, this parameter would be a comma separated list of all the hostname:port + values corresponding to them. + </para> + </section> + + <section id="bk_setupZK"> + <title>Setting up ZooKeeper</title> + <para> ZooKeeper stores metadata on behalf of BookKeeper clients and bookies. To get a minimal + ZooKeeper installation to work with BookKeeper, we can set up one server running in + standalone mode. Once we have the server running, we need to create a few znodes: + </para> + + <orderedlist> + <listitem> + <para><computeroutput> + /ledgers + </computeroutput></para> + </listitem> + + <listitem> + <para><computeroutput> + /ledgers/available + </computeroutput></para> + </listitem> + + <listitem> + <para> For each bookie, we add one znode such that the name of the znode is the + concatenation of the machine name and the port number that the bookie is + listening on. For example, if a bookie is running on bookie.foo.com an is listening + on port 3181, we add a znode + <computeroutput>/ledgers/available/bookie.foo.com:3181</computeroutput>. + </para> + </listitem> + </orderedlist> + </section> + + <section id="bk_example"> + <title>Example</title> + <para> + In the following excerpt of code, we: + </para> + + <orderedlist> + <listitem> + <para> + Create a ledger; + </para> + </listitem> + + <listitem> + <para> + Write to the ledger; + </para> + </listitem> + + <listitem> + <para> + Close the ledger; + </para> + </listitem> + + <listitem> + <para> + Open the same ledger for reading; + </para> + </listitem> + + <listitem> + <para> + Read from the ledger; + </para> + </listitem> + + <listitem> + <para> + Close the ledger again; + </para> + </listitem> + </orderedlist> + + <programlisting> +LedgerHandle lh = bkc.createLedger(ledgerPassword); +ledgerId = lh.getId(); +ByteBuffer entry = ByteBuffer.allocate(4); + +for(int i = 0; i < 10; i++){ + entry.putInt(i); + entry.position(0); + entries.add(entry.array()); + lh.addEntry(entry.array()); +} +lh.close(); +lh = bkc.openLedger(ledgerId, ledgerPassword); + +Enumeration<LedgerEntry> ls = lh.readEntries(0, 9); +int i = 0; +while(ls.hasMoreElements()){ + ByteBuffer origbb = ByteBuffer.wrap( + entries.get(i++)); + Integer origEntry = origbb.getInt(); + ByteBuffer result = ByteBuffer.wrap( + ls.nextElement().getEntry()); + + Integer retrEntry = result.getInt(); +} +lh.close(); + </programlisting> + </section> + </section> +</article> http://git-wip-us.apache.org/repos/asf/zookeeper/blob/c1efa954/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStream.xml ---------------------------------------------------------------------- diff --git a/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStream.xml b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStream.xml new file mode 100644 index 0000000..9db605a --- /dev/null +++ b/zookeeper-docs/src/documentation/content/xdocs/bookkeeperStream.xml @@ -0,0 +1,331 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Copyright 2002-2004 The Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN" +"http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd";> +<article id="bk_Stream"> + <title>Streaming with BookKeeper</title> + + <articleinfo> + <legalnotice> + <para>Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. You may + obtain a copy of the License at <ulink + url="http://www.apache.org/licenses/LICENSE-2.0";>http://www.apache.org/licenses/LICENSE-2.0</ulink>.</para> + + <para>Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an "AS IS" + BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing permissions + and limitations under the License.</para> + </legalnotice> + + <abstract> + <para>This guide contains detailed information about using how to stream bytes + on top of BookKeeper. It essentially motivates and discusses the basic stream + operations currently supported.</para> + </abstract> + </articleinfo> + <section id="bk_StreamSummary"> + <title>Summary</title> + + <para> + When using the BookKeeper API, an application has to split the data to write into entries, each + entry being a byte array. This is natural for many applications. For example, when using BookKeeper + for write-ahead logging, an application typically wants to write the modifications corresponding + to a command or a transaction. Some other applications, however, might not have a natural boundary + for entries, and may prefer to write and read streams of bytes. This is exactly the purpose of the + stream API we have implemented on top of BookKeeper. + </para> + + <para> + The stream API is implemented in the package <computeroutput>Streaming</computeroutput>, and it contains two main classes: <computeroutput>LedgerOutputStream</computeroutput> and + <computeroutput>LedgerInputStream</computeroutput>. The class names are indicative of what they do. + </para> + </section> + + <section id="bk_LedgerOutputStream"> + <title>Writing a stream of bytes</title> + <para> + Class <computeroutput>LedgerOutputStream</computeroutput> implements two constructors and five public methods: + </para> + + <para> + <computeroutput> + public LedgerOutputStream(LedgerHandle lh) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>lh</computeroutput> is a ledger handle for a previously created and open ledger. + </para> + </listitem> + </itemizedlist> + + <para> + <computeroutput> + public LedgerOutputStream(LedgerHandle lh, int size) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>lh</computeroutput> is a ledger handle for a previously created and open ledger. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>size</computeroutput> is the size of the byte buffer to store written bytes before flushing. + </para> + </listitem> + </itemizedlist> + + + <para> + <emphasis role="bold">Closing a stream.</emphasis> This call closes the stream by flushing the write buffer. + </para> + <para> + <computeroutput> + public void close() + </computeroutput> + </para> + + <para> + which has no parameters. + </para> + + <para> + <emphasis role="bold">Flushing a stream.</emphasis> This call essentially flushes the write buffer. + </para> + <para> + <computeroutput> + public synchronized void flush() + </computeroutput> + </para> + + <para> + which has no parameters. + </para> + + <para> + <emphasis role="bold">Writing bytes.</emphasis> There are three calls for writing bytes to a stream. + </para> + + <para> + <computeroutput> + public synchronized void write(byte[] b) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>b</computeroutput> is an array of bytes to write. + </para> + </listitem> + </itemizedlist> + + <para> + <computeroutput> + public synchronized void write(byte[] b, int off, int len) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>b</computeroutput> is an array of bytes to write. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>off</computeroutput> is a buffer offset. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>len</computeroutput> is the length to write. + </para> + </listitem> + </itemizedlist> + + <para> + <computeroutput> + public synchronized void write(int b) + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>b</computeroutput> contains a byte to write. The method writes the least significant byte of the integer four bytes. + </para> + </listitem> + </itemizedlist> + </section> + + <section id="bk_LedgerInputStream"> + <title>Reading a stream of bytes</title> + + <para> + Class <computeroutput>LedgerOutputStream</computeroutput> implements two constructors and four public methods: + </para> + + <para> + <computeroutput> + public LedgerInputStream(LedgerHandle lh) + throws BKException, InterruptedException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>lh</computeroutput> is a ledger handle for a previously created and open ledger. + </para> + </listitem> + </itemizedlist> + + <para> + <computeroutput> + public LedgerInputStream(LedgerHandle lh, int size) + throws BKException, InterruptedException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>lh</computeroutput> is a ledger handle for a previously created and open ledger. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>size</computeroutput> is the size of the byte buffer to store bytes that the application + will eventually read. + </para> + </listitem> + </itemizedlist> + + <para> + <emphasis role="bold">Closing.</emphasis> There is one call to close an input stream, but the call + is currently empty and the application is responsible for closing the ledger handle. + </para> + <para> + <computeroutput> + public void close() + </computeroutput> + </para> + + <para> + which has no parameters. + </para> + + <para> + <emphasis role="bold">Reading.</emphasis> There are three calls to read from the stream. + </para> + <para> + <computeroutput> + public synchronized int read() + throws IOException + </computeroutput> + </para> + + <para> + which has no parameters. + </para> + + <para> + <computeroutput> + public synchronized int read(byte[] b) + throws IOException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>b</computeroutput> is a byte array to write to. + </para> + </listitem> + </itemizedlist> + + + <para> + <computeroutput> + public synchronized int read(byte[] b, int off, int len) + throws IOException + </computeroutput> + </para> + + <para> + where: + </para> + <itemizedlist> + <listitem> + <para> + <computeroutput>b</computeroutput> is a byte array to write to. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>off</computeroutput> is an offset for byte array <computeroutput>b</computeroutput>. + </para> + </listitem> + + <listitem> + <para> + <computeroutput>len</computeroutput> is the length in bytes to write to <computeroutput>b</computeroutput>. + </para> + </listitem> + </itemizedlist> + + + </section> + </article> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/zookeeper/blob/c1efa954/zookeeper-docs/src/documentation/content/xdocs/index.xml ---------------------------------------------------------------------- diff --git a/zookeeper-docs/src/documentation/content/xdocs/index.xml b/zookeeper-docs/src/documentation/content/xdocs/index.xml new file mode 100644 index 0000000..8ed4702 --- /dev/null +++ b/zookeeper-docs/src/documentation/content/xdocs/index.xml @@ -0,0 +1,98 @@ +<?xml version="1.0"?> +<!-- + Copyright 2002-2004 The Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd";> + +<document> + + <header> + <title>ZooKeeper: Because Coordinating Distributed Systems is a Zoo</title> + </header> + + <body> + <p>ZooKeeper is a high-performance coordination service for + distributed applications. It exposes common services - such as + naming, configuration management, synchronization, and group + services - in a simple interface so you don't have to write them + from scratch. You can use it off-the-shelf to implement + consensus, group management, leader election, and presence + protocols. And you can build on it for your own, specific needs. + </p> + + <p> + The following documents describe concepts and procedures to get + you started using ZooKeeper. If you have more questions, please + ask the <a href="ext:lists">mailing list</a> or browse the + archives. + </p> + <ul> + + <li><strong>ZooKeeper Overview</strong><p>Technical Overview Documents for Client Developers, Adminstrators, and Contributors</p> + <ul><li><a href="zookeeperOver.html">Overview</a> - a bird's eye view of ZooKeeper, including design concepts and architecture</li> + <li><a href="zookeeperStarted.html">Getting Started</a> - a tutorial-style guide for developers to install, run, and program to ZooKeeper</li> + <li><a href="ext:relnotes">Release Notes</a> - new developer and user facing features, improvements, and incompatibilities</li> + </ul> + </li> + + <li><strong>Developers</strong><p> Documents for Developers using the ZooKeeper Client API</p> + <ul> + <li><a href="ext:api/index">API Docs</a> - the technical reference to ZooKeeper Client APIs</li> + <li><a href="zookeeperProgrammers.html">Programmer's Guide</a> - a client application developer's guide to ZooKeeper</li> + <li><a href="javaExample.html">ZooKeeper Java Example</a> - a simple Zookeeper client appplication, written in Java</li> + <li><a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a> - sample implementations of barriers and queues</li> + <li><a href="recipes.html">ZooKeeper Recipes</a> - higher level solutions to common problems in distributed applications</li> + </ul> + </li> + + <li><strong>Administrators & Operators</strong> <p> Documents for Administrators and Operations Engineers of ZooKeeper Deployments</p> + <ul> + <li><a href="zookeeperAdmin.html">Administrator's Guide</a> - a guide for system administrators and anyone else who might deploy ZooKeeper</li> + <li><a href="zookeeperQuotas.html">Quota Guide</a> - a guide for system administrators on Quotas in ZooKeeper. </li> + <li><a href="zookeeperJMX.html">JMX</a> - how to enable JMX in ZooKeeper</li> + <li><a href="zookeeperHierarchicalQuorums.html">Hierarchical quorums</a></li> + <li><a href="zookeeperObservers.html">Observers</a> - non-voting ensemble members that easily improve ZooKeeper's scalability</li> + </ul> + </li> + + <li><strong>Contributors</strong><p> Documents for Developers Contributing to the ZooKeeper Open Source Project</p> + <ul> + <li><a href="zookeeperInternals.html">ZooKeeper Internals</a> - assorted topics on the inner workings of ZooKeeper</li> + </ul> + </li> + + <li><strong>Miscellaneous ZooKeeper Documentation</strong> + <ul> + <li><a href="ext:wiki">Wiki</a></li> + <li><a href="ext:faq">FAQ</a></li> + </ul> + </li> + + <li><strong>BookKeeper Documentation</strong> + <p> BookKeeper is a highly-available system that implements high-performance write-ahead logging. It uses ZooKeeper for metadata, + which is the main reason for being a ZooKeeper contrib. + </p> + <ul> + <li><a href="bookkeeperOverview.html">henn, what's it again?</a></li> + <li><a href="bookkeeperStarted.html">Ok, now how do I try it out</a></li> + <li><a href="bookkeeperProgrammer.html">Awesome, but how do I integrate it with my app?</a></li> + <li><a href="bookkeeperStream.html">Can I stream bytes instead of entries?</a></li> + </ul> + </li> + </ul> + </body> + +</document>
