Repository: hbase Updated Branches: refs/heads/0.98 aff1384a5 -> 9ab53d3a2
http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/asciidoc/metrics.adoc ---------------------------------------------------------------------- diff --git a/src/main/site/asciidoc/metrics.adoc b/src/main/site/asciidoc/metrics.adoc new file mode 100644 index 0000000..91b26af --- /dev/null +++ b/src/main/site/asciidoc/metrics.adoc @@ -0,0 +1,97 @@ +//// + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +//// + += Apache HBase (TM) Metrics + +== Introduction +Apache HBase (TM) emits Hadoop link:http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/metrics/package-summary.html[metrics]. + +== Setup + +First read up on Hadoop link:http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/metrics/package-summary.html[metrics]. + +If you are using ganglia, the link:http://wiki.apache.org/hadoop/GangliaMetrics[GangliaMetrics] wiki page is useful read. + +To have HBase emit metrics, edit `$HBASE_HOME/conf/hadoop-metrics.properties` and enable metric 'contexts' per plugin. As of this writing, hadoop supports *file* and *ganglia* plugins. Yes, the hbase metrics files is named hadoop-metrics rather than _hbase-metrics_ because currently at least the hadoop metrics system has the properties filename hardcoded. Per metrics _context_, comment out the NullContext and enable one or more plugins instead. + +If you enable the _hbase_ context, on regionservers you'll see total requests since last +metric emission, count of regions and storefiles as well as a count of memstore size. +On the master, you'll see a count of the cluster's requests. + +Enabling the _rpc_ context is good if you are interested in seeing +metrics on each hbase rpc method invocation (counts and time taken). + +The _jvm_ context is useful for long-term stats on running hbase jvms -- memory used, thread counts, etc. As of this writing, if more than one jvm is running emitting metrics, at least in ganglia, the stats are aggregated rather than reported per instance. + +== Using with JMX + +In addition to the standard output contexts supported by the Hadoop +metrics package, you can also export HBase metrics via Java Management +Extensions (JMX). This will allow viewing HBase stats in JConsole or +any other JMX client. + +=== Enable HBase stats collection + +To enable JMX support in HBase, first edit `$HBASE_HOME/conf/hadoop-metrics.properties` to support metrics refreshing. (If you've running 0.94.1 and above, or have already configured `hadoop-metrics.properties` for another output context, you can skip this step). +[source,bash] +---- +# Configuration of the "hbase" context for null +hbase.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread +hbase.period=60 + +# Configuration of the "jvm" context for null +jvm.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread +jvm.period=60 + +# Configuration of the "rpc" context for null +rpc.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread +rpc.period=60 +---- + +=== Setup JMX Remote Access + +For remote access, you will need to configure JMX remote passwords and access profiles. Create the files: +`$HBASE_HOME/conf/jmxremote.passwd` (set permissions + to 600):: + +---- +monitorRole monitorpass +controlRole controlpass +---- + +`$HBASE_HOME/conf/jmxremote.access`:: + +---- +monitorRole readonly +controlRole readwrite +---- + +=== Configure JMX in HBase startup + +Finally, edit the `$HBASE_HOME/conf/hbase-env.sh` script to add JMX support: +[source,bash] +---- +HBASE_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false" +HBASE_JMX_OPTS="$HBASE_JMX_OPTS -Dcom.sun.management.jmxremote.password.file=$HBASE_HOME/conf/jmxremote.passwd" +HBASE_JMX_OPTS="$HBASE_JMX_OPTS -Dcom.sun.management.jmxremote.access.file=$HBASE_HOME/conf/jmxremote.access" + +export HBASE_MASTER_OPTS="$HBASE_JMX_OPTS -Dcom.sun.management.jmxremote.port=10101" +export HBASE_REGIONSERVER_OPTS="$HBASE_JMX_OPTS -Dcom.sun.management.jmxremote.port=10102" +---- + +After restarting the processes you want to monitor, you should now be able to run JConsole (included with the JDK since JDK 5.0) to view the statistics via JMX. HBase MBeans are exported under the *`hadoop`* domain in JMX. + + +== Understanding HBase Metrics + +For more information on understanding HBase metrics, see the link:book.html#hbase_metrics[metrics section] in the Apache HBase Reference Guide. + http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/asciidoc/old_news.adoc ---------------------------------------------------------------------- diff --git a/src/main/site/asciidoc/old_news.adoc b/src/main/site/asciidoc/old_news.adoc new file mode 100644 index 0000000..36ff510 --- /dev/null +++ b/src/main/site/asciidoc/old_news.adoc @@ -0,0 +1,117 @@ +//// + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +//// + += Old Apache HBase (TM) News + +February 10th, 2014:: link:http://www.meetup.com/hbaseusergroup/events/163139322/[HBase Meetup @ Continuuity] in Palo Alto + +January 30th, 2014:: link:http://www.meetup.com/hbaseusergroup/events/158491762/[HBase Meetup @ Apple] in Cupertino + +January 30th, 2014:: link:http://www.meetup.com/Los-Angeles-HBase-User-group/events/160560282/[Los Angeles HBase User Group] in El Segundo + +October 24th, 2013:: link:http://www.meetup.com/hbaseusergroup/events/140759692/[HBase User] and link:http://www.meetup.com/hackathon/events/144366512/[Developer] Meetup at HortonWorksin Palo Alto + +September 26, 2013:: link:http://www.meetup.com/hbaseusergroup/events/135862292/[HBase Meetup at Arista Networks] in San Francisco + +August 20th, 2013:: link:http://www.meetup.com/hbaseusergroup/events/120534362/[HBase Meetup at Flurry] in San Francisco + +July 16th, 2013:: link:http://www.meetup.com/hbaseusergroup/events/119929152/[HBase Meetup at Twitter] in San Francisco + +June 25th, 2013:: link:http://www.meetup.com/hbaseusergroup/events/119154442/[Hadoop Summit Meetup].at San Jose Convention Center + +June 14th, 2013:: link:http://kijicon.eventbrite.com/[KijiCon: Building Big Data Apps] in San Francisco. + +June 13th, 2013:: link:http://www.hbasecon.com/[HBaseCon2013] in San Francisco. Submit an Abstract! + +June 12th, 2013:: link:http://www.meetup.com/hackathon/events/123403802/[HBaseConHackAthon] at the Cloudera office in San Francisco. + +April 11th, 2013:: link:http://www.meetup.com/hbaseusergroup/events/103587852/[HBase Meetup at AdRoll] in San Francisco + +February 28th, 2013:: link:http://www.meetup.com/hbaseusergroup/events/96584102/[HBase Meetup at Intel Mission Campus] + +February 19th, 2013:: link:http://www.meetup.com/hackathon/events/103633042/[Developers PowWow] at HortonWorks' new digs + +January 23rd, 2013:: link:http://www.meetup.com/hbaseusergroup/events/91381312/[HBase Meetup at WibiData World HQ!] + +December 4th, 2012:: link:http://www.meetup.com/hackathon/events/90536432/[0.96 Bug Squashing and Testing Hackathon] at Cloudera, SF. + +October 29th, 2012:: link:http://www.meetup.com/hbaseusergroup/events/82791572/[HBase User Group Meetup] at Wize Commerce in San Mateo. + +October 25th, 2012:: link:http://www.meetup.com/HBase-NYC/events/81728932/[Strata/Hadoop World HBase Meetup.] in NYC + +September 11th, 2012:: link:http://www.meetup.com/hbaseusergroup/events/80621872/[Contributor's Pow-Wow at HortonWorks HQ.] + +August 8th, 2012:: link:http://www.apache.org/dyn/closer.cgi/hbase/[Apache HBase 0.94.1 is available for download] + +June 15th, 2012:: link:http://www.meetup.com/hbaseusergroup/events/59829652/[Birds-of-a-feather] in San Jose, day after:: link:http://hadoopsummit.org[Hadoop Summit] + +May 23rd, 2012:: link:http://www.meetup.com/hackathon/events/58953522/[HackConAthon] in Palo Alto + +May 22nd, 2012:: link:http://www.hbasecon.com[HBaseCon2012] in San Francisco + +March 27th, 2012:: link:http://www.meetup.com/hbaseusergroup/events/56021562/[Meetup @ StumbleUpon] in San Francisco + +January 19th, 2012:: link:http://www.meetup.com/hbaseusergroup/events/46702842/[Meetup @ EBay] + +January 23rd, 2012:: Apache HBase 0.92.0 released. link:http://www.apache.org/dyn/closer.cgi/hbase/[Download it!] + +December 23rd, 2011:: Apache HBase 0.90.5 released. link:http://www.apache.org/dyn/closer.cgi/hbase/[Download it!] + +November 29th, 2011:: link:http://www.meetup.com/hackathon/events/41025972/[Developer Pow-Wow in SF] at Salesforce HQ + +November 7th, 2011:: link:http://www.meetup.com/hbaseusergroup/events/35682812/[HBase Meetup in NYC (6PM)] at the AppNexus office + +August 22nd, 2011:: link:http://www.meetup.com/hbaseusergroup/events/28518471/[HBase Hackathon (11AM) and Meetup (6PM)] at FB in PA + +June 30th, 2011:: link:http://www.meetup.com/hbaseusergroup/events/20572251/[HBase Contributor Day], the day after the:: link:http://developer.yahoo.com/events/hadoopsummit2011/[Hadoop Summit] hosted by Y! + +June 8th, 2011:: link:http://berlinbuzzwords.de/wiki/hbase-workshop-and-hackathon[HBase Hackathon] in Berlin to coincide with:: link:http://berlinbuzzwords.de/[Berlin Buzzwords] + +May 19th, 2011: Apache HBase 0.90.3 released. link:http://www.apache.org/dyn/closer.cgi/hbase/[Download it!] + +April 12th, 2011: Apache HBase 0.90.2 released. link:http://www.apache.org/dyn/closer.cgi/hbase/[Download it!] + +March 21st, 2011:: link:http://www.meetup.com/hackathon/events/16770852/[HBase 0.92 Hackathon at StumbleUpon, SF] +February 22nd, 2011:: link:http://www.meetup.com/hbaseusergroup/events/16492913/[HUG12: February HBase User Group at StumbleUpon SF] +December 13th, 2010:: link:http://www.meetup.com/hackathon/calendar/15597555/[HBase Hackathon: Coprocessor Edition] +November 19th, 2010:: link:http://huguk.org/[Hadoop HUG in London] is all about Apache HBase +November 15-19th, 2010:: link:http://www.devoxx.com/display/Devoxx2K10/Home[Devoxx] features HBase Training and multiple HBase presentations + +October 12th, 2010:: HBase-related presentations by core contributors and users at:: link:http://www.cloudera.com/company/press-center/hadoop-world-nyc/[Hadoop World 2010] + +October 11th, 2010:: link:http://www.meetup.com/hbaseusergroup/calendar/14606174/[HUG-NYC: HBase User Group NYC Edition] (Night before Hadoop World) +June 30th, 2010:: link:http://www.meetup.com/hbaseusergroup/calendar/13562846/[Apache HBase Contributor Workshop] (Day after Hadoop Summit) +May 10th, 2010:: Apache HBase graduates from Hadoop sub-project to Apache Top Level Project + +April 19, 2010:: Signup for link:http://www.meetup.com/hbaseusergroup/calendar/12689490/[HBase User Group Meeting, HUG10] hosted by Trend Micro + +March 10th, 2010:: link:http://www.meetup.com/hbaseusergroup/calendar/12689351/[HBase User Group Meeting, HUG9] hosted by Mozilla + +January 27th, 2010:: Sign up for the link:http://www.meetup.com/hbaseusergroup/calendar/12241393/[HBase User Group Meeting, HUG8], at StumbleUpon in SF + +September 8th, 2010:: Apache HBase 0.20.0 is faster, stronger, slimmer, and sweeter tasting than any previous Apache HBase release. Get it off the link:http://www.apache.org/dyn/closer.cgi/hbase/[Releases] page. + +November 2-6th, 2009:: link:http://dev.us.apachecon.com/c/acus2009/[ApacheCon] in Oakland. The Apache Foundation will be celebrating its 10th anniversary in beautiful Oakland by the Bay. Lots of good talks and meetups including an HBase presentation by a couple of the lads. + +October 2nd, 2009:: HBase at Hadoop World in NYC. A few of us will be talking on Practical HBase out east at link:http://www.cloudera.com/hadoop-world-nyc[Hadoop World: NYC]. + +August 7th-9th, 2009:: HUG7 and HBase Hackathon at StumbleUpon in SF: Sign up for the:: link:http://www.meetup.com/hbaseusergroup/calendar/10950511/[HBase User Group Meeting, HUG7] or for the link:http://www.meetup.com/hackathon/calendar/10951718/[Hackathon] or for both (all are welcome!). + +June, 2009:: HBase at HadoopSummit2009 and at NOSQL: See the link:http://wiki.apache.org/hadoop/HBase/HBasePresentations[presentations] + +March 3rd, 2009 :: HUG6 -- link:http://www.meetup.com/hbaseusergroup/calendar/9764004/[HBase User Group 6] + +January 30th, 2009:: LA Hbackathon: link:http://www.meetup.com/hbasela/calendar/9450876/[HBase January Hackathon Los Angeles] at link:http://streamy.com[Streamy] in Manhattan Beach + http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/asciidoc/pseudo-distributed.adoc ---------------------------------------------------------------------- diff --git a/src/main/site/asciidoc/pseudo-distributed.adoc b/src/main/site/asciidoc/pseudo-distributed.adoc new file mode 100644 index 0000000..1eef753 --- /dev/null +++ b/src/main/site/asciidoc/pseudo-distributed.adoc @@ -0,0 +1,19 @@ +//// + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +//// + + += Running Apache HBase (TM) in pseudo-distributed mode +This page has been retired. The contents have been moved to the link:book.html#distributed[Distributed Operation: Pseudo- and Fully-distributed modes] section in the Reference Guide. + http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/asciidoc/replication.adoc ---------------------------------------------------------------------- diff --git a/src/main/site/asciidoc/replication.adoc b/src/main/site/asciidoc/replication.adoc new file mode 100644 index 0000000..0f41839 --- /dev/null +++ b/src/main/site/asciidoc/replication.adoc @@ -0,0 +1,18 @@ +//// + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +//// + += Apache HBase (TM) Replication + +This information has been moved to link:book.html#cluster_replication"[the Cluster Replication] section of the link:book.html[Apache HBase Reference Guide]. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/asciidoc/resources.adoc ---------------------------------------------------------------------- diff --git a/src/main/site/asciidoc/resources.adoc b/src/main/site/asciidoc/resources.adoc new file mode 100644 index 0000000..55af99e --- /dev/null +++ b/src/main/site/asciidoc/resources.adoc @@ -0,0 +1,22 @@ +//// + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +//// += Other Apache HBase (TM) Resources + +== Books +HBase: The Definitive Guide:: link:http://shop.oreilly.com/product/0636920014348.do[HBase: The Definitive Guide, _Random Access to Your Planet-Size Data_] by Lars George. Publisher: O'Reilly Media, Released: August 2011, Pages: 556. + +HBase In Action:: link:http://www.manning.com/dimidukkhurana[HBase In Action] By Nick Dimiduk and Amandeep Khurana. Publisher: Manning, MEAP Began: January 2012, Softbound print: Fall 2012, Pages: 350. + +HBase Administration Cookbook:: link:http://www.packtpub.com/hbase-administration-for-optimum-database-performance-cookbook/book[HBase Administration Cookbook] by Yifeng Jiang. Publisher: PACKT Publishing, Release: Expected August 2012, Pages: 335. + http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/asciidoc/sponsors.adoc ---------------------------------------------------------------------- diff --git a/src/main/site/asciidoc/sponsors.adoc b/src/main/site/asciidoc/sponsors.adoc new file mode 100644 index 0000000..56046d8 --- /dev/null +++ b/src/main/site/asciidoc/sponsors.adoc @@ -0,0 +1,30 @@ +//// + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +//// + += Apache HBase(TM) Sponsors + +First off, thanks to link:http://www.apache.org/foundation/thanks.html[all who sponsor] our parent, the Apache Software Foundation. + +The below companies have been gracious enough to provide their commerical tool offerings free of charge to the Apache HBase(TM) project. + +* The crew at link:http://www.ej-technologies.com/[ej-technologies] have been letting us use link:http://www.ej-technologies.com/products/jprofiler/overview.html[JProfiler] for years now. + +* The lads at link:http://headwaysoftware.com/[headway software] have given us a license for link:http://headwaysoftware.com/products/?code=Restructure101[Restructure101] so we can untangle our interdependency mess. + +* link:http://www.yourkit.com[YourKit] allows us to use their link:http://www.yourkit.com/overview/index.jsp[Java Profiler]. +* Some of us use link:http://www.jetbrains.com/idea[IntelliJ IDEA] thanks to link:http://www.jetbrains.com/[JetBrains]. + +== Sponsoring the Apache Software Foundation"> +To contribute to the Apache Software Foundation, a good idea in our opinion, see the link:http://www.apache.org/foundation/sponsorship.html[ASF Sponsorship] page. + http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/.htaccess ---------------------------------------------------------------------- diff --git a/src/main/site/resources/.htaccess b/src/main/site/resources/.htaccess new file mode 100644 index 0000000..20bf651 --- /dev/null +++ b/src/main/site/resources/.htaccess @@ -0,0 +1,8 @@ + +# Redirect replication URL to the right section of the book +# Rule added 2015-1-12 -- can be removed in 6 months +Redirect permanent /replication.html /book.html#_cluster_replication + +# Redirect old page-per-chapter book sections to new single file. +RedirectMatch permanent ^/book/(.*)\.html$ /book.html#$1 +RedirectMatch permanent ^/book/$ /book.html http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/book/.empty ---------------------------------------------------------------------- diff --git a/src/main/site/resources/book/.empty b/src/main/site/resources/book/.empty new file mode 100644 index 0000000..5513814 --- /dev/null +++ b/src/main/site/resources/book/.empty @@ -0,0 +1 @@ +# This directory is here so that we can have rewrite rules in our .htaccess to maintain old links. Otherwise we fall under some top-level niceness redirects because we have a file named book.html. http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/css/site.css ---------------------------------------------------------------------- diff --git a/src/main/site/resources/css/site.css b/src/main/site/resources/css/site.css index f26d03c..17f0ff0 100644 --- a/src/main/site/resources/css/site.css +++ b/src/main/site/resources/css/site.css @@ -72,8 +72,10 @@ h4 { #banner { background: none; + padding: 10px; } +/* #banner img { padding: 10px; margin: auto; @@ -82,6 +84,7 @@ h4 { float: center; height:; } + */ #breadcrumbs { background-image: url(); http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/architecture.gif ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/architecture.gif b/src/main/site/resources/images/architecture.gif new file mode 100644 index 0000000..8d84a23 Binary files /dev/null and b/src/main/site/resources/images/architecture.gif differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/bc_basic.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/bc_basic.png b/src/main/site/resources/images/bc_basic.png new file mode 100644 index 0000000..231de93 Binary files /dev/null and b/src/main/site/resources/images/bc_basic.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/bc_config.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/bc_config.png b/src/main/site/resources/images/bc_config.png new file mode 100644 index 0000000..53250cf Binary files /dev/null and b/src/main/site/resources/images/bc_config.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/bc_l1.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/bc_l1.png b/src/main/site/resources/images/bc_l1.png new file mode 100644 index 0000000..36d7e55 Binary files /dev/null and b/src/main/site/resources/images/bc_l1.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/bc_l2_buckets.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/bc_l2_buckets.png b/src/main/site/resources/images/bc_l2_buckets.png new file mode 100644 index 0000000..5163928 Binary files /dev/null and b/src/main/site/resources/images/bc_l2_buckets.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/bc_stats.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/bc_stats.png b/src/main/site/resources/images/bc_stats.png new file mode 100644 index 0000000..d8c6384 Binary files /dev/null and b/src/main/site/resources/images/bc_stats.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/big_h_logo.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/big_h_logo.png b/src/main/site/resources/images/big_h_logo.png new file mode 100644 index 0000000..5256094 Binary files /dev/null and b/src/main/site/resources/images/big_h_logo.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/coprocessor_stats.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/coprocessor_stats.png b/src/main/site/resources/images/coprocessor_stats.png new file mode 100644 index 0000000..2fc8703 Binary files /dev/null and b/src/main/site/resources/images/coprocessor_stats.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/data_block_diff_encoding.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/data_block_diff_encoding.png b/src/main/site/resources/images/data_block_diff_encoding.png new file mode 100644 index 0000000..0bd03a4 Binary files /dev/null and b/src/main/site/resources/images/data_block_diff_encoding.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/data_block_no_encoding.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/data_block_no_encoding.png b/src/main/site/resources/images/data_block_no_encoding.png new file mode 100644 index 0000000..56498b4 Binary files /dev/null and b/src/main/site/resources/images/data_block_no_encoding.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/data_block_prefix_encoding.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/data_block_prefix_encoding.png b/src/main/site/resources/images/data_block_prefix_encoding.png new file mode 100644 index 0000000..4271847 Binary files /dev/null and b/src/main/site/resources/images/data_block_prefix_encoding.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/favicon.ico ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/favicon.ico b/src/main/site/resources/images/favicon.ico new file mode 100644 index 0000000..6e4d0f7 Binary files /dev/null and b/src/main/site/resources/images/favicon.ico differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/hadoop-logo.jpg ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/hadoop-logo.jpg b/src/main/site/resources/images/hadoop-logo.jpg new file mode 100644 index 0000000..809525d Binary files /dev/null and b/src/main/site/resources/images/hadoop-logo.jpg differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/hbase_replication_diagram.jpg ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/hbase_replication_diagram.jpg b/src/main/site/resources/images/hbase_replication_diagram.jpg new file mode 100644 index 0000000..c110309 Binary files /dev/null and b/src/main/site/resources/images/hbase_replication_diagram.jpg differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/hbasecon2015.30percent.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/hbasecon2015.30percent.png b/src/main/site/resources/images/hbasecon2015.30percent.png new file mode 100644 index 0000000..26896a4 Binary files /dev/null and b/src/main/site/resources/images/hbasecon2015.30percent.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/hfile.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/hfile.png b/src/main/site/resources/images/hfile.png new file mode 100644 index 0000000..5762970 Binary files /dev/null and b/src/main/site/resources/images/hfile.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/hfilev2.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/hfilev2.png b/src/main/site/resources/images/hfilev2.png new file mode 100644 index 0000000..54cc0cf Binary files /dev/null and b/src/main/site/resources/images/hfilev2.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/jumping-orca_rotated.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/jumping-orca_rotated.png b/src/main/site/resources/images/jumping-orca_rotated.png new file mode 100644 index 0000000..4c2c72e Binary files /dev/null and b/src/main/site/resources/images/jumping-orca_rotated.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/jumping-orca_rotated.xcf ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/jumping-orca_rotated.xcf b/src/main/site/resources/images/jumping-orca_rotated.xcf new file mode 100644 index 0000000..01be6ff Binary files /dev/null and b/src/main/site/resources/images/jumping-orca_rotated.xcf differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/jumping-orca_rotated_12percent.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/jumping-orca_rotated_12percent.png b/src/main/site/resources/images/jumping-orca_rotated_12percent.png new file mode 100644 index 0000000..1942f9a Binary files /dev/null and b/src/main/site/resources/images/jumping-orca_rotated_12percent.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/jumping-orca_rotated_25percent.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/jumping-orca_rotated_25percent.png b/src/main/site/resources/images/jumping-orca_rotated_25percent.png new file mode 100644 index 0000000..219c657 Binary files /dev/null and b/src/main/site/resources/images/jumping-orca_rotated_25percent.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/region_states.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/region_states.png b/src/main/site/resources/images/region_states.png new file mode 100644 index 0000000..ba69e97 Binary files /dev/null and b/src/main/site/resources/images/region_states.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/replication_overview.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/replication_overview.png b/src/main/site/resources/images/replication_overview.png new file mode 100644 index 0000000..47d7b4c Binary files /dev/null and b/src/main/site/resources/images/replication_overview.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/resources/images/timeline_consistency.png ---------------------------------------------------------------------- diff --git a/src/main/site/resources/images/timeline_consistency.png b/src/main/site/resources/images/timeline_consistency.png new file mode 100644 index 0000000..94c47e0 Binary files /dev/null and b/src/main/site/resources/images/timeline_consistency.png differ http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/site.xml ---------------------------------------------------------------------- diff --git a/src/main/site/site.xml b/src/main/site/site.xml index 0d60e00..81c9315 100644 --- a/src/main/site/site.xml +++ b/src/main/site/site.xml @@ -19,18 +19,34 @@ */ --> - <project xmlns="http://maven.apache.org/DECORATION/1.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/DECORATION/1.0.0 http://maven.apache.org/xsd/decoration-1.0.0.xsd";> + <skin> + <groupId>lt.velykis.maven.skins</groupId> + <artifactId>reflow-maven-skin</artifactId> + <version>1.1.1</version> + </skin> + <custom> + <reflowSkin> + <theme>bootswatch-spacelab</theme> + <bottomNav maxSpan="9"> + <column>Apache HBase Project</column> + <column>^Documentation</column> + <column>0.94 Documentation|ASF</column> + </bottomNav> + </reflowSkin> + </custom> <bannerLeft> <name>Apache HBase</name> <src>images/hbase_logo.png</src> <href>http://hbase.apache.org/</href> </bannerLeft> - <bannerRight /> - <version position="right" /> - <publishDate position="right" /> + <bannerRight> + <name>Apache HBase Orca</name> + <src>images/hbasecon2015.30percent.png</src> + <href>http://hbasecon.com/</href> + </bannerRight> <body> <menu name="Apache HBase Project"> <item name="Overview" href="index.html"/> @@ -47,35 +63,30 @@ <item name="Other resources" href="resources.html" /> </menu> <menu name="Documentation"> - <item name="Getting Started" href="book/quickstart.html" /> + <item name="Reference Guide" href="book.html" target="_blank" /> + <item name="Getting Started" href="book.html#quickstart" /> <item name="User API" href="apidocs/index.html" /> <item name="Developer API" href="devapidocs/index.html" /> <item name="X-Ref" href="xref/index.html" /> - <item name="Ref Guide (multi-page)" href="book/book.html" /> - <item name="Ref Guide (single-page)" href="book.html" /> <item name="ä¸æåèæå(å页)" href="http://abloz.com/hbase/book.html"; /> - <item name="FAQ" href="book/faq.html" /> + <item name="FAQ" href="book.html#faq" /> <item name="Videos/Presentations" href="book.html#other.info" /> <item name="Wiki" href="http://wiki.apache.org/hadoop/Hbase"; /> <item name="ACID Semantics" href="acid-semantics.html" /> <item name="Bulk Loads" href="book.html#arch.bulk.load" /> - <item name="Metrics" href="metrics.html" /> - <item name="HBase on Windows" href="cygwin.html" /> - <item name="Cluster replication" href="replication.html" /> + <item name="Metrics" href="metrics.html" /> + <item name="HBase on Windows" href="cygwin.html" /> + <item name="Cluster replication" href="replication.html" /> </menu> <menu name="0.94 Documentation"> - <item name="API" href="0.94/apidocs/index.html" /> - <item name="X-Ref" href="0.94/xref/index.html" /> - <item name="Ref Guide (single-page)" href="0.94/book.html" /> + <item name="API" href="0.94/apidocs/index.html" /> + <item name="X-Ref" href="0.94/xref/index.html" /> + <item name="Ref Guide (single-page)" href="0.94/book.html" /> </menu> <menu name="ASF"> - <item name="Apache Software Foundation" href="http://www.apache.org/foundation/"; /> - <item name="How Apache Works" href="http://www.apache.org/foundation/how-it-works.html"; /> - <item name="Sponsoring Apache" href="http://www.apache.org/foundation/sponsorship.html"; /> + <item name="Apache Software Foundation" href="http://www.apache.org/foundation/"; /> + <item name="How Apache Works" href="http://www.apache.org/foundation/how-it-works.html"; /> + <item name="Sponsoring Apache" href="http://www.apache.org/foundation/sponsorship.html"; /> </menu> </body> - <skin> - <groupId>org.apache.maven.skins</groupId> - <artifactId>maven-stylus-skin</artifactId> - </skin> </project> http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/xdoc/export_control.xml ---------------------------------------------------------------------- diff --git a/src/main/site/xdoc/export_control.xml b/src/main/site/xdoc/export_control.xml new file mode 100644 index 0000000..0bd00e1 --- /dev/null +++ b/src/main/site/xdoc/export_control.xml @@ -0,0 +1,55 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" + "http://forrest.apache.org/dtd/document-v20.dtd";> + +<document xmlns="http://maven.apache.org/XDOC/2.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/XDOC/2.0 http://maven.apache.org/xsd/xdoc-2.0.xsd";> + <properties> + <title> + Export Control + </title> + </properties> + <body> + <section name="Export Control"> +<p> +This distribution uses or includes cryptographic software. The country in +which you currently reside may have restrictions on the import, possession, +use, and/or re-export to another country, of encryption software. BEFORE +using any encryption software, please check your country's laws, regulations +and policies concerning the import, possession, or use, and re-export of +encryption software, to see if this is permitted. See the +<a href="http://www.wassenaar.org/";>Wassenaar Arrangement</a> for more +information.</p> +<p> +The U.S. Government Department of Commerce, Bureau of Industry and Security +(BIS), has classified this software as Export Commodity Control Number (ECCN) +5D002.C.1, which includes information security software using or performing +cryptographic functions with asymmetric algorithms. The form and manner of this +Apache Software Foundation distribution makes it eligible for export under the +License Exception ENC Technology Software Unrestricted (TSU) exception (see the +BIS Export Administration Regulations, Section 740.13) for both object code and +source code.</p> +<p> +Apache HBase uses the built-in java cryptography libraries. See Oracle's +information regarding +<a href="http://www.oracle.com/us/products/export/export-regulations-345813.html";>Java cryptographic export regulations</a> +for more details.</p> + </section> + </body> +</document> http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/xdoc/index.xml ---------------------------------------------------------------------- diff --git a/src/main/site/xdoc/index.xml b/src/main/site/xdoc/index.xml index aa971f2..a40ab4b 100644 --- a/src/main/site/xdoc/index.xml +++ b/src/main/site/xdoc/index.xml @@ -29,7 +29,7 @@ <p> Use Apache HBase when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. -Apache HBase is an open-source, distributed, versioned, column-oriented store modeled after Google's <a href="http://research.google.com/archive/bigtable.html";>Bigtable: A Distributed Storage System for Structured Data</a> by Chang et al. +Apache HBase is an open-source, distributed, versioned, non-relational database modeled after Google's <a href="http://research.google.com/archive/bigtable.html";>Bigtable: A Distributed Storage System for Structured Data</a> by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, Apache HBase provides Bigtable-like capabilities on top of Hadoop and HDFS. </p> <h4>Features</h4> @@ -63,17 +63,23 @@ Apache HBase is an open-source, distributed, versioned, column-oriented store mo <p>See the <a href="http://hbase.apache.org/book/architecture.html#arch.overview";>Architecture Overview</a>, the <a href="http://hbase.apache.org/book/faq.html";>Apache HBase Reference Guide FAQ</a>, and the other documentation links on the left! </p> + <h4>Export Control</h4> + <p>The HBase distribution includes cryptographic software. See the export control notice <a href="export_control.html">here</a> + </p> </section> <section name="News"> - <p>October 24th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/140759692/";>HBase User and <a href="http://www.meetup.com/hackathon/events/144366512/";>Developer</a> Meetup at HortonWorks</a>.in Palo Alto</p> - <p>September 26, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/135862292/";>HBase Meetup at Arista Networks</a>.in San Francisco</p> - <p>August 20th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/120534362/";>HBase Meetup at Flurry</a>.in San Francisco</p> - <p>July 16th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/119929152/";>HBase Meetup at Twitter</a>.in San Francisco</p> - <p>June 25th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/119154442/";>Hadoop Summit Meetup</a>.at San Jose Convention Center</p> - <p>June 14th, 2013 <a href="http://kijicon.eventbrite.com/";>KijiCon: Building Big Data Apps</a> in San Francisco.</p> - <p>June 13th, 2013 <a href="http://www.hbasecon.com/";>HBaseCon2013</a> in San Francisco. Submit an Abstract!</p> - <p>June 12th, 2013 <a href="http://www.meetup.com/hackathon/events/123403802/";>HBaseConHackAthon</a> at the Cloudera office in San Francisco.</p> - + <p>May 7th, 2015 <a href="http://hbasecon.com/";>HBaseCon2015</a> in San Francisco</p> + <p>February 17th, 2015 <a href="http://www.meetup.com/hbaseusergroup/events/219260093/";>HBase meetup around Strata+Hadoop World</a> in San Jose</p> + <p>January 15th, 2015 <a href="http://www.meetup.com/hbaseusergroup/events/218744798/";>HBase meetup @ AppDynamics</a> in San Francisco</p> + <p>November 20th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/205219992/";>HBase meetup @ WANdisco</a> in San Ramon</p> + <p>October 27th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/207386102/";>HBase Meetup @ Apple</a> in Cupertino</p> + <p>October 15th, 2014 <a href="http://www.meetup.com/HBase-NYC/events/207655552";>HBase Meetup @ Google</a> on the night before Strata/HW in NYC</p> + <p>September 25th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/203173692/";>HBase Meetup @ Continuuity</a> in Palo Alto</p> + <p>August 28th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/197773762/";>HBase Meetup @ Sift Science</a> in San Francisco</p> + <p>July 17th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/190994082/";>HBase Meetup @ HP</a> in Sunnyvale</p> + <p>June 5th, 2014 <a href="http://www.meetup.com/Hadoop-Summit-Community-San-Jose/events/179081342/";>HBase BOF at Hadoop Summit</a>, San Jose Convention Center</p> + <p>May 5th, 2014 <a href="http://www.hbasecon.com";>HBaseCon2014</a> at the Hilton San Francisco on Union Square</p> + <p>March 12th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/160757912/";>HBase Meetup @ Ancestry.com</a> in San Francisco</p> <p><small><a href="old_news.html">Old News</a></small></p> </section> </body> http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/xdoc/old_news.xml ---------------------------------------------------------------------- diff --git a/src/main/site/xdoc/old_news.xml b/src/main/site/xdoc/old_news.xml index 343d370..20138e8 100644 --- a/src/main/site/xdoc/old_news.xml +++ b/src/main/site/xdoc/old_news.xml @@ -27,6 +27,17 @@ </properties> <body> <section name="Old News"> + <p>February 10th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/163139322/";>HBase Meetup @ Continuuity</a> in Palo Alto</p> + <p>January 30th, 2014 <a href="http://www.meetup.com/hbaseusergroup/events/158491762/";>HBase Meetup @ Apple</a> in Cupertino</p> + <p>January 30th, 2014 <a href="http://www.meetup.com/Los-Angeles-HBase-User-group/events/160560282/";>Los Angeles HBase User Group</a> in El Segundo</p> + <p>October 24th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/140759692/";>HBase User and <a href="http://www.meetup.com/hackathon/events/144366512/";>Developer</a> Meetup at HortonWorks</a>.in Palo Alto</p> + <p>September 26, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/135862292/";>HBase Meetup at Arista Networks</a>.in San Francisco</p> + <p>August 20th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/120534362/";>HBase Meetup at Flurry</a>.in San Francisco</p> + <p>July 16th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/119929152/";>HBase Meetup at Twitter</a>.in San Francisco</p> + <p>June 25th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/119154442/";>Hadoop Summit Meetup</a>.at San Jose Convention Center</p> + <p>June 14th, 2013 <a href="http://kijicon.eventbrite.com/";>KijiCon: Building Big Data Apps</a> in San Francisco.</p> + <p>June 13th, 2013 <a href="http://www.hbasecon.com/";>HBaseCon2013</a> in San Francisco. Submit an Abstract!</p> + <p>June 12th, 2013 <a href="http://www.meetup.com/hackathon/events/123403802/";>HBaseConHackAthon</a> at the Cloudera office in San Francisco.</p> <p>April 11th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/103587852/";>HBase Meetup at AdRoll</a> in San Francisco</p> <p>February 28th, 2013 <a href="http://www.meetup.com/hbaseusergroup/events/96584102/";>HBase Meetup at Intel Mission Campus</a></p> <p>February 19th, 2013 <a href="http://www.meetup.com/hackathon/events/103633042/";>Developers PowWow</a> at HortonWorks' new digs</p> http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/site/xdoc/replication.xml ---------------------------------------------------------------------- diff --git a/src/main/site/xdoc/replication.xml b/src/main/site/xdoc/replication.xml index 7b11773..2633f08 100644 --- a/src/main/site/xdoc/replication.xml +++ b/src/main/site/xdoc/replication.xml @@ -26,533 +26,6 @@ </title> </properties> <body> - <section name="Overview"> - <p> - The replication feature of Apache HBase (TM) provides a way to copy data between HBase deployments. It - can serve as a disaster recovery solution and can contribute to provide - higher availability at the HBase layer. It can also serve more practically; - for example, as a way to easily copy edits from a web-facing cluster to a "MapReduce" - cluster which will process old and new data and ship back the results - automatically. - </p> - <p> - The basic architecture pattern used for Apache HBase replication is (HBase cluster) master-push; - it is much easier to keep track of whatâs currently being replicated since - each region server has its own write-ahead-log (aka WAL or HLog), just like - other well known solutions like MySQL master/slave replication where - thereâs only one bin log to keep track of. One master cluster can - replicate to any number of slave clusters, and each region server will - participate to replicate their own stream of edits. For more information - on the different properties of master/slave replication and other types - of replication, please consult <a href="http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html";> - How Google Serves Data From Multiple Datacenters</a>. - </p> - <p> - The replication is done asynchronously, meaning that the clusters can - be geographically distant, the links between them can be offline for - some time, and rows inserted on the master cluster wonât be - available at the same time on the slave clusters (eventual consistency). - </p> - <p> - The replication format used in this design is conceptually the same as - <a href="http://dev.mysql.com/doc/refman/5.1/en/replication-formats.html";> - MySQLâs statement-based replication </a>. Instead of SQL statements, whole - WALEdits (consisting of multiple cell inserts coming from the clients' - Put and Delete) are replicated in order to maintain atomicity. - </p> - <p> - The HLogs from each region server are the basis of HBase replication, - and must be kept in HDFS as long as they are needed to replicate data - to any slave cluster. Each RS reads from the oldest log it needs to - replicate and keeps the current position inside ZooKeeper to simplify - failure recovery. That position can be different for every slave - cluster, same for the queue of HLogs to process. - </p> - <p> - The clusters participating in replication can be of asymmetric sizes - and the master cluster will do its âbest effortâ to balance the stream - of replication on the slave clusters by relying on randomization. - </p> - <p> - As of version 0.92, Apache HBase supports master/master and cyclic - replication as well as replication to multiple slaves. - </p> - <img src="images/replication_overview.png"/> - </section> - <section name="Enabling replication"> - <p> - The guide on enabling and using cluster replication is contained - in the API documentation shipped with your Apache HBase distribution. - </p> - <p> - The most up-to-date documentation is - <a href="apidocs/org/apache/hadoop/hbase/replication/package-summary.html#requirements"> - available at this address</a>. - </p> - </section> - <section name="Life of a log edit"> - <p> - The following sections describe the life of a single edit going from a - client that communicates with a master cluster all the way to a single - slave cluster. - </p> - <section name="Normal processing"> - <p> - The client uses an API that sends a Put, Delete or ICV to a region - server. The key values are transformed into a WALEdit by the region - server and is inspected by the replication code that, for each family - that is scoped for replication, adds the scope to the edit. The edit - is appended to the current WAL and is then applied to its MemStore. - </p> - <p> - In a separate thread, the edit is read from the log (as part of a batch) - and only the KVs that are replicable are kept (that is, that they are part - of a family scoped GLOBAL in the family's schema, non-catalog so not - .META. or -ROOT-, and did not originate in the target slave cluster - in - case of cyclic replication). - </p> - <p> - The edit is then tagged with the master's cluster UUID. - When the buffer is filled, or the reader hits the end of the file, - the buffer is sent to a random region server on the slave cluster. - </p> - <p> - Synchronously, the region server that receives the edits reads them - sequentially and separates each of them into buffers, one per table. - Once all edits are read, each buffer is flushed using HTable, the normal - HBase client.The master's cluster UUID is retained in the edits applied at - the slave cluster in order to allow cyclic replication. - </p> - <p> - Back in the master cluster's region server, the offset for the current - WAL that's being replicated is registered in ZooKeeper. - </p> - </section> - <section name="Non-responding slave clusters"> - <p> - The edit is inserted in the same way. - </p> - <p> - In the separate thread, the region server reads, filters and buffers - the log edits the same way as during normal processing. The slave - region server that's contacted doesn't answer to the RPC, so the master - region server will sleep and retry up to a configured number of times. - If the slave RS still isn't available, the master cluster RS will select a - new subset of RS to replicate to and will retry sending the buffer of - edits. - </p> - <p> - In the mean time, the WALs will be rolled and stored in a queue in - ZooKeeper. Logs that are archived by their region server (archiving is - basically moving a log from the region server's logs directory to a - central logs archive directory) will update their paths in the in-memory - queue of the replicating thread. - </p> - <p> - When the slave cluster is finally available, the buffer will be applied - the same way as during normal processing. The master cluster RS will then - replicate the backlog of logs. - </p> - </section> - </section> - <section name="Internals"> - <p> - This section describes in depth how each of replication's internal - features operate. - </p> - <section name="Replication Zookeeper State"> - <p> - HBase replication maintains all of its state in Zookeeper. By default, this state is - contained in the base znode: - </p> - <pre> - /hbase/replication - </pre> - <p> - There are three major child znodes in the base replication znode: - <ul> - <li><b>State znode:</b> /hbase/replication/state</li> - <li><b>Peers znode:</b> /hbase/replication/peers</li> - <li><b>RS znode:</b> /hbase/replication/rs</li> - </ul> - </p> - <section name="The State znode"> - <p> - The <b>state znode</b> indicates whether or not replication is enabled on the cluster - corresponding to this zookeeper quorum. It does not have any child znodes and simply - contains a boolean value. This value is initialized on startup based on the - <i>hbase.replication</i> config parameter in the <i>hbase-site.xml</i> file. The status - value is read/maintained by the <i>ReplicationZookeeper.ReplicationStatusTracker</i> - class. It is also cached locally using an AtomicBoolean in the <i>ReplicationZookeeper</i> - class. This value can be changed on a live cluster using the <i>stop_replication</i> - command available through the hbase shell. - </p> - <pre> - /hbase/replication/state [VALUE: true] - </pre> - </section> - <section name="The Peers znode"> - <p> - The <b>peers znode</b> contains a list of all peer replication clusters and the - current replication state of those clusters. It has one child <i>peer znode</i> - for each peer cluster. The <i>peer znode</i> is named with the cluster id provided - by the user in the HBase shell. The value of the <i>peer znode</i> contains - the peers cluster key provided by the user in the HBase Shell. The cluster key - contains a list of zookeeper nodes in the clusters quorum, the client port for the - zookeeper quorum, and the base znode for HBase - (i.e. âzk1.host.com,zk2.host.com,zk3.host.com:2181:/hbaseâ). - </p> - <pre> - /hbase/replication/peers - /1 [Value: zk1.host.com,zk2.host.com,zk3.host.com:2181:/hbase] - /2 [Value: zk5.host.com,zk6.host.com,zk7.host.com:2181:/hbase] - </pre> - <p> - Each of these <i>peer znodes</i> has a child znode that indicates whether or not - replication is enabled on that peer cluster. These <i>peer-state znodes</i> do not - have child znodes and simply contain a boolean value (i.e. ENABLED or DISABLED). - This value is read/maintained by the <i>ReplicationPeer.PeerStateTracker</i> class. - It is also cached locally using an AtomicBoolean in the <i>ReplicationPeer</i> class. - </p> - <pre> - /hbase/replication/peers - /1/peer-state [Value: ENABLED] - /2/peer-state [Value: DISABLED] - </pre> - </section> - <section name="The RS znode"> - <p> - The <b>rs znode</b> contains a list of all outstanding HLog files in the cluster - that need to be replicated. The list is divided into a set of queues organized by - region server and the peer cluster the region server is shipping the HLogs to. The - <b>rs znode</b> has one child znode for each region server in the cluster. The child - znode name is simply the regionserver name (a concatenation of the region serverâs - hostname, client port and start code). These region servers could either be dead or alive. - </p> - <pre> - /hbase/replication/rs - /hostname.example.org,6020,1234 - /hostname2.example.org,6020,2856 - </pre> - <p> - Within each region server znode, the region server maintains a set of HLog replication - queues. Each region server has one queue for every peer cluster it replicates to. - These queues are represented by child znodes named using the cluster id of the peer - cluster they represent (see the peer znode section). - </p> - <pre> - /hbase/replication/rs - /hostname.example.org,6020,1234 - /1 - /2 - </pre> - <p> - Each queue has one child znode for every HLog that still needs to be replicated. - The value of these HLog child znodes is the latest position that has been replicated. - This position is updated every time a HLog entry is replicated. - </p> - <pre> - /hbase/replication/rs - /hostname.example.org,6020,1234 - /1 - 23522342.23422 [VALUE: 254] - 12340993.22342 [VALUE: 0] - </pre> - </section> - </section> - <section name="Configuration Parameters"> - <section name="Zookeeper znode paths"> - <p> - All of the base znode names are configurable through parameters: - </p> - <table border="1"> - <tr> - <td><b>Parameter</b></td> - <td><b>Default Value</b></td> - </tr> - <tr> - <td>zookeeper.znode.parent</td> - <td>/hbase</td> - </tr> - <tr> - <td>zookeeper.znode.replication</td> - <td>replication</td> - </tr> - <tr> - <td>zookeeper.znode.replication.peers</td> - <td>peers</td> - </tr> - <tr> - <td>zookeeper.znode.replication.peers.state</td> - <td>peer-state</td> - </tr> - <tr> - <td>zookeeper.znode.replication.rs</td> - <td>rs</td> - </tr> - </table> - <p> - The default replication znode structure looks like the following: - </p> - <pre> - /hbase/replication/state - /hbase/replication/peers/{peerId}/peer-state - /hbase/replication/rs - </pre> - </section> - <section name="Other parameters"> - <ul> - <li><b>hbase.replication</b> (Default: false) - Controls whether replication is enabled - or disabled for the cluster.</li> - <li><b>replication.sleep.before.failover</b> (Default: 2000) - The amount of time a failover - worker waits before attempting to replicate a dead region serverâs HLog queues.</li> - <li><b>replication.executor.workers</b> (Default: 1) - The number of dead region servers - one region server should attempt to failover simultaneously.</li> - </ul> - </section> - </section> - <section name="Choosing region servers to replicate to"> - <p> - When a master cluster RS initiates a replication source to a slave cluster, - it first connects to the slave's ZooKeeper ensemble using the provided - cluster key (that key is composed of the value of hbase.zookeeper.quorum, - zookeeper.znode.parent and hbase.zookeeper.property.clientPort). It - then scans the "rs" directory to discover all the available sinks - (region servers that are accepting incoming streams of edits to replicate) - and will randomly choose a subset of them using a configured - ratio (which has a default value of 10%). For example, if a slave - cluster has 150 machines, 15 will be chosen as potential recipient for - edits that this master cluster RS will be sending. Since this is done by all - master cluster RSs, the probability that all slave RSs are used is very high, - and this method works for clusters of any size. For example, a master cluster - of 10 machines replicating to a slave cluster of 5 machines with a ratio - of 10% means that the master cluster RSs will choose one machine each - at random, thus the chance of overlapping and full usage of the slave - cluster is higher. - </p> - <p> - A ZK watcher is placed on the ${zookeeper.znode.parent}/rs node of - the slave cluster by each of the master cluster's region servers. - This watch is used to monitor changes in the composition of the - slave cluster. When nodes are removed from the slave cluster (or - if nodes go down and/or come back up), the master cluster's region - servers will respond by selecting a new pool of slave region servers - to replicate to. - </p> - </section> - <section name="Keeping track of logs"> - <p> - Every master cluster RS has its own znode in the replication znodes hierarchy. - It contains one znode per peer cluster (if 5 slave clusters, 5 znodes - are created), and each of these contain a queue - of HLogs to process. Each of these queues will track the HLogs created - by that RS, but they can differ in size. For example, if one slave - cluster becomes unavailable for some time then the HLogs should not be deleted, - thus they need to stay in the queue (while the others are processed). - See the section named "Region server failover" for an example. - </p> - <p> - When a source is instantiated, it contains the current HLog that the - region server is writing to. During log rolling, the new file is added - to the queue of each slave cluster's znode just before it's made available. - This ensures that all the sources are aware that a new log exists - before HLog is able to append edits into it, but this operations is - now more expensive. - The queue items are discarded when the replication thread cannot read - more entries from a file (because it reached the end of the last block) - and that there are other files in the queue. - This means that if a source is up-to-date and replicates from the log - that the region server writes to, reading up to the "end" of the - current file won't delete the item in the queue. - </p> - <p> - When a log is archived (because it's not used anymore or because there's - too many of them per hbase.regionserver.maxlogs typically because insertion - rate is faster than region flushing), it will notify the source threads that the path - for that log changed. If the a particular source was already done with - it, it will just ignore the message. If it's in the queue, the path - will be updated in memory. If the log is currently being replicated, - the change will be done atomically so that the reader doesn't try to - open the file when it's already moved. Also, moving a file is a NameNode - operation so, if the reader is currently reading the log, it won't - generate any exception. - </p> - </section> - <section name="Reading, filtering and sending edits"> - <p> - By default, a source will try to read from a log file and ship log - entries as fast as possible to a sink. This is first limited by the - filtering of log entries; only KeyValues that are scoped GLOBAL and - that don't belong to catalog tables will be retained. A second limit - is imposed on the total size of the list of edits to replicate per slave, - which by default is 64MB. This means that a master cluster RS with 3 slaves - will use at most 192MB to store data to replicate. This doesn't account - the data filtered that wasn't garbage collected. - </p> - <p> - Once the maximum size of edits was buffered or the reader hits the end - of the log file, the source thread will stop reading and will choose - at random a sink to replicate to (from the list that was generated by - keeping only a subset of slave RSs). It will directly issue a RPC to - the chosen machine and will wait for the method to return. If it's - successful, the source will determine if the current file is emptied - or if it should continue to read from it. If the former, it will delete - the znode in the queue. If the latter, it will register the new offset - in the log's znode. If the RPC threw an exception, the source will retry - 10 times until trying to find a different sink. - </p> - </section> - <section name="Cleaning logs"> - <p> - If replication isn't enabled, the master's logs cleaning thread will - delete old logs using a configured TTL. This doesn't work well with - replication since archived logs passed their TTL may still be in a - queue. Thus, the default behavior is augmented so that if a log is - passed its TTL, the cleaning thread will lookup every queue until it - finds the log (while caching the ones it finds). If it's not found, - the log will be deleted. The next time it has to look for a log, - it will first use its cache. - </p> - </section> - <section name="Region server failover"> - <p> - As long as region servers don't fail, keeping track of the logs in ZK - doesn't add any value. Unfortunately, they do fail, so since ZooKeeper - is highly available we can count on it and its semantics to help us - managing the transfer of the queues. - </p> - <p> - All the master cluster RSs keep a watcher on every other one of them to be - notified when one dies (just like the master does). When it happens, - they all race to create a znode called "lock" inside the dead RS' znode - that contains its queues. The one that creates it successfully will - proceed by transferring all the queues to its own znode (one by one - since ZK doesn't support the rename operation) and will delete all the - old ones when it's done. The recovered queues' znodes will be named - with the id of the slave cluster appended with the name of the dead - server. - </p> - <p> - Once that is done, the master cluster RS will create one new source thread per - copied queue, and each of them will follow the read/filter/ship pattern. - The main difference is that those queues will never have new data since - they don't belong to their new region server, which means that when - the reader hits the end of the last log, the queue's znode will be - deleted and the master cluster RS will close that replication source. - </p> - <p> - For example, consider a master cluster with 3 region servers that's - replicating to a single slave with id '2'. The following hierarchy - represents what the znodes layout could be at some point in time. We - can see the RSs' znodes all contain a "peers" znode that contains a - single queue. The znode names in the queues represent the actual file - names on HDFS in the form "address,port.timestamp". - </p> - <pre> -/hbase/replication/rs/ - 1.1.1.1,60020,123456780/ - 2/ - 1.1.1.1,60020.1234 (Contains a position) - 1.1.1.1,60020.1265 - 1.1.1.2,60020,123456790/ - 2/ - 1.1.1.2,60020.1214 (Contains a position) - 1.1.1.2,60020.1248 - 1.1.1.2,60020.1312 - 1.1.1.3,60020, 123456630/ - 2/ - 1.1.1.3,60020.1280 (Contains a position) - </pre> - <p> - Now let's say that 1.1.1.2 loses its ZK session. The survivors will race - to create a lock, and for some reasons 1.1.1.3 wins. It will then start - transferring all the queues to its local peers znode by appending the - name of the dead server. Right before 1.1.1.3 is able to clean up the - old znodes, the layout will look like the following: - </p> - <pre> -/hbase/replication/rs/ - 1.1.1.1,60020,123456780/ - 2/ - 1.1.1.1,60020.1234 (Contains a position) - 1.1.1.1,60020.1265 - 1.1.1.2,60020,123456790/ - lock - 2/ - 1.1.1.2,60020.1214 (Contains a position) - 1.1.1.2,60020.1248 - 1.1.1.2,60020.1312 - 1.1.1.3,60020,123456630/ - 2/ - 1.1.1.3,60020.1280 (Contains a position) - - 2-1.1.1.2,60020,123456790/ - 1.1.1.2,60020.1214 (Contains a position) - 1.1.1.2,60020.1248 - 1.1.1.2,60020.1312 - </pre> - <p> - Some time later, but before 1.1.1.3 is able to finish replicating the - last HLog from 1.1.1.2, let's say that it dies too (also some new logs - were created in the normal queues). The last RS will then try to lock - 1.1.1.3's znode and will begin transferring all the queues. The new - layout will be: - </p> - <pre> -/hbase/replication/rs/ - 1.1.1.1,60020,123456780/ - 2/ - 1.1.1.1,60020.1378 (Contains a position) - - 2-1.1.1.3,60020,123456630/ - 1.1.1.3,60020.1325 (Contains a position) - 1.1.1.3,60020.1401 - - 2-1.1.1.2,60020,123456790-1.1.1.3,60020,123456630/ - 1.1.1.2,60020.1312 (Contains a position) - 1.1.1.3,60020,123456630/ - lock - 2/ - 1.1.1.3,60020.1325 (Contains a position) - 1.1.1.3,60020.1401 - - 2-1.1.1.2,60020,123456790/ - 1.1.1.2,60020.1312 (Contains a position) - </pre> - </section> - </section> - <section name="Replication Metrics"> - Following the some useful metrics which can be used to check the replication progress: - <ul> - <li><b>source.sizeOfLogQueue:</b> number of HLogs to process (excludes the one which is being - processed) at the Replication source</li> - <li><b>source.shippedOps:</b> number of mutations shipped</li> - <li><b>source.logEditsRead:</b> number of mutations read from HLogs at the replication source</li> - <li><b>source.ageOfLastShippedOp:</b> age of last batch that was shipped by the replication source</li> - </ul> - Please note that the above metrics are at the global level at this regionserver. In 0.95.0 and onwards, these - metrics are also exposed per peer level. - </section> - - <section name="FAQ"> - <section name="GLOBAL means replicate? Any provision to replicate only to cluster X and not to cluster Y? or is that for later?"> - <p> - Yes, this is for much later. - </p> - </section> - <section name="You need a bulk edit shipper? Something that allows you transfer 64MB of edits in one go?"> - <p> - You can use the HBase-provided utility called CopyTable from the package - org.apache.hadoop.hbase.mapreduce in order to have a discp-like tool to - bulk copy data. - </p> - </section> - <section name="Is it a mistake that WALEdit doesn't carry Put and Delete objects, that we have to reinstantiate not only when replicating but when replaying edits also?"> - <p> - Yes, this behavior would help a lot but it's not currently available - in HBase (BatchUpdate had that, but it was lost in the new API). - </p> - </section> - </section> + <p>This information has been moved to <a href="http://hbase.apache.org/book.html#cluster_replication";>the Cluster Replication</a> section of the <a href="http://hbase.apache.org/book.html";>Apache HBase Reference Guide</a>.</p> </body> </document> http://git-wip-us.apache.org/repos/asf/hbase/blob/7bf6c024/src/main/xslt/configuration_to_asciidoc_chapter.xsl ---------------------------------------------------------------------- diff --git a/src/main/xslt/configuration_to_asciidoc_chapter.xsl b/src/main/xslt/configuration_to_asciidoc_chapter.xsl new file mode 100644 index 0000000..428d95a --- /dev/null +++ b/src/main/xslt/configuration_to_asciidoc_chapter.xsl @@ -0,0 +1,92 @@ +<?xml version="1.0"?> +<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"; version="1.0"> + + +<!-- +/** + * Copyright 2010 The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +This stylesheet is used making an html version of hbase-default.adoc. +--> + +<xsl:output method="text"/> + +<!-- Normalize space --> +<xsl:template match="text()"> + <xsl:if test="normalize-space(.)"> + <xsl:value-of select="normalize-space(.)"/> + </xsl:if> +</xsl:template> + +<!-- Grab nodes of the <configuration> element --> +<xsl:template match="configuration"> + +<!-- Print the license at the top of the file --> +//// +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +//// +:doctype: book +:numbered: +:toc: left +:icons: font +:experimental: + +[[hbase_default_configurations]] +=== HBase Default Configuration + +The documentation below is generated using the default hbase configuration file, _hbase-default.xml_, as source. + + <xsl:for-each select="property"> + <xsl:if test="not(@skipInDoc)"> +[[<xsl:apply-templates select="name"/>]] +`<xsl:apply-templates select="name"/>`:: ++ +.Description +<xsl:apply-templates select="description"/> ++ +.Default +<xsl:choose> + <xsl:when test="value != ''">`<xsl:apply-templates select="value"/>` + +</xsl:when> + <xsl:otherwise>none</xsl:otherwise> +</xsl:choose> + </xsl:if> + </xsl:for-each> + +</xsl:template> +</xsl:stylesheet>
