HBASE-20142 Copy master doc into branch-2 and edit to make it suit 2.0.0

Copy from master of the content at src/main back to branch-2.0.

I then removed backup and spark files from _chapter subdir and
applied HBASE-17918, doc on serial replication, in reverse.

Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/9ef75b96
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/9ef75b96
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/9ef75b96

Branch: refs/heads/branch-2.0
Commit: 9ef75b96d196ed9169189626808e191187487f37
Parents: e47eb5b
Author: Michael Stack <st...@apache.org>
Authored: Mon Apr 9 20:30:18 2018 -0700
Committer: Michael Stack <st...@apache.org>
Committed: Mon Apr 9 20:56:28 2018 -0700

 .../appendix_contributing_to_documentation.adoc |   6 +-
 .../_chapters/appendix_hfile_format.adoc        |   2 +-
 src/main/asciidoc/_chapters/architecture.adoc   | 109 ++-
 src/main/asciidoc/_chapters/backup_restore.adoc | 912 -------------------
 src/main/asciidoc/_chapters/community.adoc      |  51 +-
 src/main/asciidoc/_chapters/compression.adoc    |   2 +-
 src/main/asciidoc/_chapters/configuration.adoc  |  73 +-
 src/main/asciidoc/_chapters/datamodel.adoc      |  35 +
 src/main/asciidoc/_chapters/developer.adoc      |  80 +-
 .../asciidoc/_chapters/getting_started.adoc     |  68 +-
 src/main/asciidoc/_chapters/hbase-default.adoc  |   2 +-
 src/main/asciidoc/_chapters/mapreduce.adoc      |   2 +-
 src/main/asciidoc/_chapters/ops_mgt.adoc        | 203 ++++-
 src/main/asciidoc/_chapters/schema_design.adoc  |  31 +-
 src/main/asciidoc/_chapters/shell.adoc          |   8 +-
 src/main/asciidoc/_chapters/tracing.adoc        |   6 +-
 .../asciidoc/_chapters/troubleshooting.adoc     | 191 ++--
 src/main/asciidoc/_chapters/unit_testing.adoc   |   2 -
 src/main/asciidoc/_chapters/upgrading.adoc      | 514 ++++++-----
 src/main/asciidoc/book.adoc                     |   1 -
 20 files changed, 806 insertions(+), 1492 deletions(-)

diff --git 
index 6570c9c..a603c16 100644
--- a/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
+++ b/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
@@ -119,7 +119,7 @@ JIRA and add a version number to the name of the new patch.
 === Editing the HBase Website
-The source for the HBase website is in the HBase source, in the 
_src/main/site/_ directory.
+The source for the HBase website is in the HBase source, in the _src/site/_ 
 Within this directory, source for the individual pages is in the _xdocs/_ 
 and images referenced in those pages are in the _resources/images/_ directory.
 This directory also stores images used in the HBase Reference Guide.
@@ -216,7 +216,7 @@ link:http://www.google.com[Google]
 image::sunset.jpg[Alt Text]
-(put the image in the src/main/site/resources/images directory)
+(put the image in the src/site/resources/images directory)
 | An inline image | The image with alt text, as part of the text flow |
 image:sunset.jpg [Alt Text]
@@ -389,7 +389,7 @@ Inline images cannot have titles. They are generally small 
images like GUI butto
 image:sunset.jpg[Alt Text]
-When doing a local build, save the image to the 
_src/main/site/resources/images/_ directory.
+When doing a local build, save the image to the _src/site/resources/images/_ 
 When you link to the image, do not include the directory portion of the path.
 The image will be copied to the appropriate target location during the build 
of the output.

diff --git a/src/main/asciidoc/_chapters/appendix_hfile_format.adoc 
index 18eafe6..20f46d3 100644
--- a/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
+++ b/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
@@ -94,7 +94,7 @@ The version of HBase introducing the above features reads 
both version 1 and 2 H
 A version 2 HFile is structured as follows:
 .HFile Version 2 Structure
-image:hfilev2.png[HFile Version 2]
+image::hfilev2.png[HFile Version 2]
 ==== Unified version 2 block format

diff --git a/src/main/asciidoc/_chapters/architecture.adoc 
index b29244c..bc29d4b 100644
--- a/src/main/asciidoc/_chapters/architecture.adoc
+++ b/src/main/asciidoc/_chapters/architecture.adoc
@@ -951,8 +951,11 @@ However, if a RegionServer crashes or becomes unavailable 
before the MemStore is
 If writing to the WAL fails, the entire operation to modify the data fails.
 HBase uses an implementation of the 
-Usually, there is only one instance of a WAL per RegionServer.
-The RegionServer records Puts and Deletes to it, before recording them to the 
<<store.memstore>> for the affected <<store>>.
+Usually, there is only one instance of a WAL per RegionServer. An exception
+is the RegionServer that is carrying _hbase:meta_; the _meta_ table gets its
+own dedicated WAL.
+The RegionServer records Puts and Deletes to its WAL, before recording them
+these Mutations <<store.memstore>> for the affected <<store>>.
 .The HLog
@@ -962,9 +965,30 @@ In 0.94, HLog was the name of the implementation of the 
 You will likely find references to the HLog in documentation tailored to these 
older versions.
-The WAL resides in HDFS in the _/hbase/WALs/_ directory (prior to HBase 0.94, 
they were stored in _/hbase/.logs/_), with subdirectories per region.
+The WAL resides in HDFS in the _/hbase/WALs/_ directory, with subdirectories 
per region.
+For more general information about the concept of write ahead logs, see the 
+link:http://en.wikipedia.org/wiki/Write-ahead_logging[Write-Ahead Log] article.
+==== WAL Providers
+In HBase, there are a number of WAL imlementations (or 'Providers'). Each is 
+by a short name label (that unfortunately is not always descriptive). You set 
the provider in
+_hbase-site.xml_ passing the WAL provder short-name as the value on the
+_hbase.wal.provider_ property (Set the provider for _hbase:meta_ using the
+_hbase.wal.meta_provider_ property).
+ * _asyncfs_: The *default*. New since hbase-2.0.0 (HBASE-15536, HBASE-14790). 
This _AsyncFSWAL_ provider, as it identifies itself in RegionServer logs, is 
built on a new non-blocking dfsclient implementation. It is currently resident 
in the hbase codebase but intent is to move it back up into HDFS itself. WALs 
edits are written concurrently ("fan-out") style to each of the WAL-block 
replicas on each DataNode rather than in a chained pipeline as the default 
client does. Latencies should be better. See 
 HBase Improements and Practices at Xiaomi] at slide 14 onward for more detail 
on implementation.
+ * _filesystem_: This was the default in hbase-1.x releases. It is built on 
the blocking _DFSClient_ and writes to replicas in classic _DFSCLient_ pipeline 
mode. In logs it identifies as _FSHLog_ or _FSHLogProvider_.
+ * _multiwal_: This provider is made of multiple instances of _asyncfs_ or  
_filesystem_. See the next section for more on _multiwal_.
+Look for the lines like the below in the RegionServer log to see which 
provider is in place (The below shows the default AsyncFSWALProvider):
+2018-04-02 13:22:37,983 INFO  [regionserver/ve0528:16020] wal.WALFactory: 
Instantiating WALProvider of type class 
-For more general information about the concept of write ahead logs, see the 
Wikipedia link:http://en.wikipedia.org/wiki/Write-ahead_logging[Write-Ahead 
Log] article.
 ==== MultiWAL
 With a single WAL per RegionServer, the RegionServer must write to the WAL 
serially, because HDFS files must be sequential. This causes the WAL to be a 
performance bottleneck.
@@ -1090,28 +1114,28 @@ The general process for log splitting, as described in 
 . If distributed log processing is enabled, the HMaster creates a _split log 
manager_ instance when the cluster is started.
   .. The split log manager manages all log files which need to be scanned and 
-  .. The split log manager places all the logs into the ZooKeeper splitlog 
node (_/hbase/splitlog_) as tasks.
-  .. You can view the contents of the splitlog by issuing the following 
`zkCli` command. Example output is shown.
+  .. The split log manager places all the logs into the ZooKeeper splitWAL 
node (_/hbase/splitWAL_) as tasks.
+  .. You can view the contents of the splitWAL by issuing the following 
`zkCli` command. Example output is shown.
-ls /hbase/splitlog
+ls /hbase/splitWAL
 The output contains some non-ASCII characters.
 When decoded, it looks much more simple:
@@ -1122,7 +1146,7 @@ The listing represents WAL file names to be scanned and 
split, which is a list o
 The split log manager is responsible for the following ongoing tasks:
-* Once the split log manager publishes all the tasks to the splitlog znode, it 
monitors these task nodes and waits for them to be processed.
+* Once the split log manager publishes all the tasks to the splitWAL znode, it 
monitors these task nodes and waits for them to be processed.
 * Checks to see if there are any dead split log workers queued up.
   If it finds tasks claimed by unresponsive workers, it will resubmit those 
   If the resubmit fails due to some ZooKeeper exception, the dead worker is 
queued up again for retry.
@@ -1140,7 +1164,7 @@ The split log manager is responsible for the following 
ongoing tasks:
   In the example output below, the first line of the output shows that the 
task is currently unassigned.
 unassigned host2.sample.com:57000
 cZxid = 0×7115
@@ -1171,12 +1195,12 @@ Based on the state of the task whose data is changed, 
the split log manager does
 Each RegionServer runs a daemon thread called the _split log worker_, which 
does the work to split the logs.
 The daemon thread starts when the RegionServer starts, and registers itself to 
watch HBase znodes.
-If any splitlog znode children change, it notifies a sleeping worker thread to 
wake up and grab more tasks.
+If any splitWAL znode children change, it notifies a sleeping worker thread to 
wake up and grab more tasks.
 If a worker's current task's node data is changed,
 the worker checks to see if the task has been taken by another worker.
 If so, the worker thread stops work on the current task.
-The worker monitors the splitlog znode constantly.
+The worker monitors the splitWAL znode constantly.
 When a new task appears, the split log worker retrieves the task paths and 
checks each one until it finds an unclaimed task, which it attempts to claim.
 If the claim was successful, it attempts to perform the task and updates the 
task's `state` property based on the splitting outcome.
 At this point, the split log worker scans for another unclaimed task.
@@ -1219,21 +1243,17 @@ A possible downside to WAL compression is that we lose 
more data from the last b
 mid-write. If entries in this last block were added with new dictionary 
entries but we failed persist the amended
 dictionary because of an abrupt termination, a read of this last block may not 
be able to resolve last-written entries.
-==== WAL Compression ====
+==== Durability
+It is possible to set _durability_ on each Mutation or on a Table basis. 
Options include:
-The content of the WAL can be compressed using LRU Dictionary compression.
-This can be used to speed up WAL replication to different datanodes.
-The dictionary can store up to 2^15^ elements; eviction starts after this 
number is exceeded.
-To enable WAL compression, set the `hbase.regionserver.wal.enablecompression` 
property to `true`.
-The default value for this property is `false`.
-By default, WAL tag compression is turned on when WAL compression is enabled.
-You can turn off WAL tag compression by setting the 
`hbase.regionserver.wal.tags.enablecompression` property to 'false'.
+ * _SKIP_WAL_: Do not write Mutations to the WAL (See the next section, 
+ * _ASYNC_WAL_: Write the WAL asynchronously; do not hold-up clients waiting 
on the sync of their write to the filesystem but return immediately; the 
Mutation will be flushed to the WAL at a later time. This option currently may 
lose data. See HBASE-16689.
+ * _SYNC_WAL_: The *default*. Each edit is sync'd to HDFS before we return 
success to the client.
+ * _FSYNC_WAL_: Each edit is fsync'd to HDFS and the filesystem before we 
return success to the client.
-A possible downside to WAL compression is that we lose more data from the last 
block in the WAL if it ill-terminated
-mid-write. If entries in this last block were added with new dictionary 
entries but we failed persist the amended
-dictionary because of an abrupt termination, a read of this last block may not 
be able to resolve last-written entries. 
+Do not confuse the _ASYNC_WAL_ option on a Mutation or Table with the 
_AsyncFSWAL_ writer; they are distinct
+options unfortunately closely named
 ==== Disabling the WAL
@@ -1249,6 +1269,7 @@ There is no way to disable the WAL for only a specific 
 WARNING: If you disable the WAL for anything other than bulk loads, your data 
is at risk.
 == Regions
@@ -1509,11 +1530,14 @@ Alphanumeric Rowkeys::
 Using a Custom Algorithm::
   The RegionSplitter tool is provided with HBase, and uses a _SplitAlgorithm_ 
to determine split points for you.
   As parameters, you give it the algorithm, desired number of regions, and 
column families.
-  It includes two split algorithms.
+  It includes three split algorithms.
   The first is the
   algorithm, which assumes the row keys are hexadecimal strings.
-  The second,
+  The second is the
+  algorithm, which assumes the row keys are decimal strings in the range 
00000000 to 99999999.
+  The third,
   assumes the row keys are random byte arrays.
   You will probably need to develop your own
@@ -1609,10 +1633,10 @@ Type the following to see usage:
 $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.io.hfile.HFile
-For example, to view the content of the file 
_hdfs://, type 
the following:
+For example, to view the content of the file 
 type the following:
- $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f 
+ $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f 
 If you leave off the option -v to see just a summary on the HFile.
 See usage for other things to do with the `HFile` tool.
@@ -1770,9 +1794,20 @@ These parameters will be explained in context, and then 
will be given in a table
 ====== Being Stuck
 When the MemStore gets too large, it needs to flush its contents to a 
-However, a Store can only have `hbase.hstore.blockingStoreFiles` files, so the 
MemStore needs to wait for the number of StoreFiles to be reduced by one or 
more compactions.
-However, if the MemStore grows larger than 
`hbase.hregion.memstore.flush.size`, it is not able to flush its contents to a 
-If the MemStore is too large and the number of StoreFiles is also too high, 
the algorithm is said to be "stuck". The compaction algorithm checks for this 
"stuck" situation and provides mechanisms to alleviate it.
+However, Stores are configured with a bound on the number StoreFiles,
+`hbase.hstore.blockingStoreFiles`, and if in excess, the MemStore flush must 
+until the StoreFile count is reduced by one or more compactions. If the 
+is too large and the number of StoreFiles is also too high, the algorithm is 
+to be "stuck". By default we'll wait on compactions up to
+`hbase.hstore.blockingWaitTime` milliseconds. If this period expires, we'll 
+anyways even though we are in excess of the
+`hbase.hstore.blockingStoreFiles` count.
+Upping the `hbase.hstore.blockingStoreFiles` count will allow flushes to happen
+but a Store with many StoreFiles in will likely have higher read latencies. 
Try to
+figure why Compactions are not keeping up. Is it a write spurt that is bringing
+about this situation or is a regular occurance and the cluster is 
+for the volume of writes?
 ====== The ExploringCompactionPolicy Algorithm

diff --git a/src/main/asciidoc/_chapters/backup_restore.adoc 
deleted file mode 100644
index a9dbcf5..0000000
--- a/src/main/asciidoc/_chapters/backup_restore.adoc
+++ /dev/null
@@ -1,912 +0,0 @@
- *
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-= Backup and Restore
-:doctype: book
-:toc: left
-:icons: font
-== Overview
-Backup and restore is a standard operation provided by many databases. An 
effective backup and restore
-strategy helps ensure that users can recover data in case of unexpected 
failures. The HBase backup and restore
-feature helps ensure that enterprises using HBase as a canonical data 
repository can recover from catastrophic
-failures. Another important feature is the ability to restore the database to 
a particular
-point-in-time, commonly referred to as a snapshot.
-The HBase backup and restore feature provides the ability to create full 
backups and incremental backups on
-tables in an HBase cluster. The full backup is the foundation on which 
incremental backups are applied
-to build iterative snapshots. Incremental backups can be run on a schedule to 
capture changes over time,
-for example by using a Cron task. Incremental backups are more cost-effective 
than full backups because they only capture
-the changes since the last backup and they also enable administrators to 
restore the database to any prior incremental backup. Furthermore, the
-utilities also enable table-level data backup-and-recovery if you do not want 
to restore the entire dataset
-of the backup.
-The backup and restore feature supplements the HBase Replication feature. 
While HBase replication is ideal for
-creating "hot" copies of the data (where the replicated data is immediately 
available for query), the backup and
-restore feature is ideal for creating "cold" copies of data (where a manual 
step must be taken to restore the system).
-Previously, users only had the ability to create full backups via the 
ExportSnapshot functionality. The incremental
-backup implementation is the novel improvement over the previous "art" 
provided by ExportSnapshot.
-== Terminology
-The backup and restore feature introduces new terminology which can be used to 
understand how control flows through the
-* _A backup_: A logical unit of data and metadata which can restore a table to 
its state at a specific point in time.
-* _Full backup_: a type of backup which wholly encapsulates the contents of 
the table at a point in time.
-* _Incremental backup_: a type of backup which contains the changes in a table 
since a full backup.
-* _Backup set_: A user-defined name which references one or more tables over 
which a backup can be executed.
-* _Backup ID_: A unique names which identifies one backup from the rest, e.g. 
-== Planning
-There are some common strategies which can be used to implement backup and 
restore in your environment. The following section
-shows how these strategies are implemented and identifies potential tradeoffs 
with each.
-WARNING: This backup and restore tools has not been tested on Transparent Data 
Encryption (TDE) enabled HDFS clusters.
-This is related to the open issue 
-=== Backup within a cluster
-This strategy stores the backups on the same cluster as where the backup was 
taken. This approach is only appropriate for testing
-as it does not provide any additional safety on top of what the software 
itself already provides.
-.Intra-Cluster Backup
-=== Backup using a dedicated cluster
-This strategy provides greater fault tolerance and provides a path towards 
disaster recovery. In this setting, you will
-store the backup on a separate HDFS cluster by supplying the backup 
destination cluster’s HDFS URL to the backup utility.
-You should consider backing up to a different physical location, such as a 
different data center.
-Typically, a backup-dedicated HDFS cluster uses a more economical hardware 
profile to save money.
-.Dedicated HDFS Cluster Backup
-=== Backup to the Cloud or a storage vendor appliance
-Another approach to safeguarding HBase incremental backups is to store the 
data on provisioned, secure servers that belong
-to third-party vendors and that are located off-site. The vendor can be a 
public cloud provider or a storage vendor who uses
-a Hadoop-compatible file system, such as S3 and other HDFS-compatible 
-.Backup to Cloud or Vendor Storage Solutions
-NOTE: The HBase backup utility does not support backup to multiple 
destinations. A workaround is to manually create copies
-of the backup files from HDFS or S3.
-== First-time configuration steps
-This section contains the necessary configuration changes that must be made in 
order to use the backup and restore feature.
-As this feature makes significant use of YARN's MapReduce framework to 
parallelize these I/O heavy operations, configuration
-changes extend outside of just `hbase-site.xml`.
-=== Allow the "hbase" system user in YARN
-The YARN *container-executor.cfg* configuration file must have the following 
property setting: _allowed.system.users=hbase_. No spaces
-are allowed in entries of this configuration file.
-WARNING: Skipping this step will result in runtime errors when executing the 
first backup tasks.
-*Example of a valid container-executor.cfg file for backup and restore:*
-=== HBase specific changes
-Add the following properties to hbase-site.xml and restart HBase if it is 
already running.
-NOTE: The ",..." is an ellipsis meant to imply that this is a comma-separated 
list of values, not literal text which should be added to hbase-site.xml.
-  <name>hbase.backup.enable</name>
-  <value>true</value>
-  <name>hbase.master.logcleaner.plugins</name>
-  <value>org.apache.hadoop.hbase.backup.master.BackupLogCleaner,...</value>
-  <name>hbase.procedure.master.classes</name>
-  <name>hbase.procedure.regionserver.classes</name>
-  <name>hbase.coprocessor.region.classes</name>
-  <value>org.apache.hadoop.hbase.backup.BackupObserver,...</value>
-  <name>hbase.master.hfilecleaner.plugins</name>
-  <value>org.apache.hadoop.hbase.backup.BackupHFileCleaner,...</value>
-== Backup and Restore commands
-This covers the command-line utilities that administrators would run to 
create, restore, and merge backups. Tools to
-inspect details on specific backup sessions is covered in the next section, 
<<br.administration,Administration of Backup Images>>.
-Run the command `hbase backup help <command>` to access the online help that 
provides basic information about a command
-and its options. The below information is captured in this help message for 
each command.
-// hbase backup create
-### Creating a Backup Image
-For HBase clusters also using Apache Phoenix: include the SQL system catalog 
tables in the backup. In the event that you
-need to restore the HBase backup, access to the system catalog tables enable 
you to resume Phoenix interoperability with the
-restored data.
-The first step in running the backup and restore utilities is to perform a 
full backup and to store the data in a separate image
-from the source. At a minimum, you must do this to get a baseline before you 
can rely on incremental backups.
-Run the following command as HBase superuser:
-hbase backup create <type> <backup_path>
-After the command finishes running, the console prints a SUCCESS or FAILURE 
status message. The SUCCESS message includes a _backup_ ID.
-The backup ID is the Unix time (also known as Epoch time) that the HBase 
master received the backup request from the client.
-Record the backup ID that appears at the end of a successful backup. In case 
the source cluster fails and you need to recover the
-dataset with a restore operation, having the backup ID readily available can 
save time.
-#### Positional Command-Line Arguments
-  The type of backup to execute: _full_ or _incremental_. As a reminder, an 
_incremental_ backup requires a _full_ backup to
-  already exist.
-  The _backup_path_ argument specifies the full filesystem URI of where to 
store the backup image. Valid prefixes are
-  are _hdfs:_, _webhdfs:_, _gpfs:_, and _s3fs:_.
-#### Named Command-Line Arguments
-_-t <table_name[,table_name]>_::
-  A comma-separated list of tables to back up. If no tables are specified, all 
tables are backed up. No regular-expression or
-  wildcard support is present; all table names must be explicitly listed. See 
<<br.using.backup.sets,Backup Sets>> for more
-  information about peforming operations on collections of tables. Mutually 
exclusive with the _-s_ option; one of these
-  named options are required.
-_-s <backup_set_name>_::
-  Identify tables to backup based on a backup set. See 
<<br.using.backup.sets,Using Backup Sets>> for the purpose and usage
-  of backup sets. Mutually exclusive with the _-t_ option.
-_-w <number_workers>_::
-  (Optional) Specifies the number of parallel workers to copy data to backup 
destination. Backups are currently executed by MapReduce jobs
-  so this value corresponds to the number of Mappers that will be spawned by 
the job.
-_-b <bandwidth_per_worker>_::
-  (Optional) Specifies the bandwidth of each worker in MB per second.
-  (Optional) Enables "DEBUG" mode which prints additional logging about the 
backup creation.
-_-q <name>_::
-  (Optional) Allows specification of the name of a YARN queue which the 
MapReduce job to create the backup should be executed in. This option
-  is useful to prevent backup tasks from stealing resources away from other 
MapReduce jobs of high importance.
-#### Example usage
-$ hbase backup create full hdfs://host5:8020/data/backup -t SALES2,SALES3 -w 3
-This command creates a full backup image of two tables, SALES2 and SALES3, in 
the HDFS instance who NameNode is host5:8020
-in the path _/data/backup_. The _-w_ option specifies that no more than three 
parallel works complete the operation.
-// hbase backup restore
-### Restoring a Backup Image
-Run the following command as an HBase superuser. You can only restore a backup 
on a running HBase cluster because the data must be
-redistributed the RegionServers for the operation to complete successfully.
-hbase restore <backup_path> <backup_id>
-#### Positional Command-Line Arguments
-  The _backup_path_ argument specifies the full filesystem URI of where to 
store the backup image. Valid prefixes are
-  are _hdfs:_, _webhdfs:_, _gpfs:_, and _s3fs:_.
-  The backup ID that uniquely identifies the backup image to be restored.
-#### Named Command-Line Arguments
-_-t <table_name[,table_name]>_::
-  A comma-separated list of tables to restore. See 
<<br.using.backup.sets,Backup Sets>> for more
-  information about peforming operations on collections of tables. Mutually 
exclusive with the _-s_ option; one of these
-  named options are required.
-_-s <backup_set_name>_::
-  Identify tables to backup based on a backup set. See 
<<br.using.backup.sets,Using Backup Sets>> for the purpose and usage
-  of backup sets. Mutually exclusive with the _-t_ option.
-_-q <name>_::
-  (Optional) Allows specification of the name of a YARN queue which the 
MapReduce job to create the backup should be executed in. This option
-  is useful to prevent backup tasks from stealing resources away from other 
MapReduce jobs of high importance.
-  (Optional) Perform a dry-run of the restore. The actions are checked, but 
not executed.
-_-m <target_tables>_::
-  (Optional) A comma-separated list of tables to restore into. If this option 
is not provided, the original table name is used. When
-  this option is provided, there must be an equal number of entries provided 
in the `-t` option.
-  (Optional) Overwrites the target table for the restore if the table already 
-#### Example of Usage
-hbase backup restore /tmp/backup_incremental backupId_1467823988425 -t 
-This command restores two tables of an incremental backup image. In this 
-• `/tmp/backup_incremental` is the path to the directory containing the 
backup image.
-• `backupId_1467823988425` is the backup ID.
-• `mytable1` and `mytable2` are the names of tables in the backup image to 
be restored.
-// hbase backup merge
-### Merging Incremental Backup Images
-This command can be used to merge two or more incremental backup images into a 
single incremental
-backup image. This can be used to consolidate multiple, small incremental 
backup images into a single
-larger incremental backup image. This command could be used to merge hourly 
incremental backups
-into a daily incremental backup image, or daily incremental backups into a 
weekly incremental backup.
-$ hbase backup merge <backup_ids>
-#### Positional Command-Line Arguments
-  A comma-separated list of incremental backup image IDs that are to be 
combined into a single image.
-#### Named Command-Line Arguments
-#### Example usage
-$ hbase backup merge backupId_1467823988425,backupId_1467827588425
-// hbase backup set
-### Using Backup Sets
-Backup sets can ease the administration of HBase data backups and restores by 
reducing the amount of repetitive input
-of table names. You can group tables into a named backup set with the `hbase 
backup set add` command. You can then use
-the -set option to invoke the name of a backup set in the `hbase backup 
create` or `hbase backup restore` rather than list
-individually every table in the group. You can have multiple backup sets.
-NOTE: Note the differentiation between the `hbase backup set add` command and 
the _-set_ option. The `hbase backup set add`
-command must be run before using the `-set` option in a different command 
because backup sets must be named and defined
-before using backup sets as a shortcut.
-If you run the `hbase backup set add` command and specify a backup set name 
that does not yet exist on your system, a new set
-is created. If you run the command with the name of an existing backup set 
name, then the tables that you specify are added
-to the set.
-In this command, the backup set name is case-sensitive.
-NOTE: The metadata of backup sets are stored within HBase. If you do not have 
access to the original HBase cluster with the
-backup set metadata, then you must specify individual table names to restore 
the data.
-To create a backup set, run the following command as the HBase superuser:
-$ hbase backup set <subcommand> <backup_set_name> <tables>
-#### Backup Set Subcommands
-The following list details subcommands of the hbase backup set command.
-NOTE: You must enter one (and no more than one) of the following subcommands 
after hbase backup set to complete an operation.
-Also, the backup set name is case-sensitive in the command-line utility.
-  Adds table[s] to a backup set. Specify a _backup_set_name_ value after this 
argument to create a backup set.
-  Removes tables from the set. Specify the tables to remove in the tables 
-  Lists all backup sets.
-  Displays a description of a backup set. The information includes whether the 
set has full
-  or incremental backups, start and end times of the backups, and a list of 
the tables in the set. This subcommand must precede
-  a valid value for the _backup_set_name_ value.
-  Deletes a backup set. Enter the value for the _backup_set_name_ option 
directly after the `hbase backup set delete` command.
-#### Positional Command-Line Arguments
-  Use to assign or invoke a backup set name. The backup set name must contain 
only printable characters and cannot have any spaces.
-  List of tables (or a single table) to include in the backup set. Enter the 
table names as a comma-separated list. If no tables
-  are specified, all tables are included in the set.
-TIP: Maintain a log or other record of the case-sensitive backup set names and 
the corresponding tables in each set on a separate
-or remote cluster, backup strategy. This information can help you in case of 
failure on the primary cluster.
-#### Example of Usage
-$ hbase backup set add Q1Data TEAM3,TEAM_4
-Depending on the environment, this command results in _one_ of the following 
-* If the `Q1Data` backup set does not exist, a backup set containing tables 
`TEAM_3` and `TEAM_4` is created.
-* If the `Q1Data` backup set exists already, the tables `TEAM_3` and `TEAM_4` 
are added to the `Q1Data` backup set.
-## Administration of Backup Images
-The `hbase backup` command has several subcommands that help with 
administering backup images as they accumulate. Most production
-environments require recurring backups, so it is necessary to have utilities 
to help manage the data of the backup repository.
-Some subcommands enable you to find information that can help identify backups 
that are relevant in a search for particular data.
-You can also delete backup images.
-The following list details each `hbase backup subcommand` that can help 
administer backups. Run the full command-subcommand line as
-the HBase superuser.
-// hbase backup progress
-### Managing Backup Progress
-You can monitor a running backup in another terminal session by running the 
_hbase backup progress_ command and specifying the backup ID as an argument.
-For example, run the following command as hbase superuser to view the progress 
of a backup
-$ hbase backup progress <backup_id>
-#### Positional Command-Line Arguments
-  Specifies the backup that you want to monitor by seeing the progress 
information. The backupId is case-sensitive.
-#### Named Command-Line Arguments
-#### Example usage
-hbase backup progress backupId_1467823988425
-// hbase backup history
-### Managing Backup History
-This command displays a log of backup sessions. The information for each 
session includes backup ID, type (full or incremental), the tables
-in the backup, status, and start and end time. Specify the number of backup 
sessions to display with the optional -n argument.
-$ hbase backup history <backup_id>
-#### Positional Command-Line Arguments
-  Specifies the backup that you want to monitor by seeing the progress 
information. The backupId is case-sensitive.
-#### Named Command-Line Arguments
-_-n <num_records>_::
-  (Optional) The maximum number of backup records (Default: 10).
-_-p <backup_root_path>_::
-  The full filesystem URI of where backup images are stored.
-_-s <backup_set_name>_::
-  The name of the backup set to obtain history for. Mutually exclusive with 
the _-t_ option.
-_-t_ <table_name>::
-  The name of table to obtain history for. Mutually exclusive with the _-s_ 
-#### Example usage
-$ hbase backup history
-$ hbase backup history -n 20
-$ hbase backup history -t WebIndexRecords
-// hbase backup describe
-### Describing a Backup Image
-This command can be used to obtain information about a specific backup image.
-$ hbase backup describe <backup_id>
-#### Positional Command-Line Arguments
-  The ID of the backup image to describe.
-#### Named Command-Line Arguments
-#### Example usage
-$ hbase backup describe backupId_1467823988425
-// hbase backup delete
-### Deleting a Backup Image
-This command can be used to delete a backup image which is no longer needed.
-$ hbase backup delete <backup_id>
-#### Positional Command-Line Arguments
-  The ID to the backup image which should be deleted.
-#### Named Command-Line Arguments
-#### Example usage
-$ hbase backup delete backupId_1467823988425
-// hbase backup repair
-### Backup Repair Command
-This command attempts to correct any inconsistencies in persisted backup 
metadata which exists as
-the result of software errors or unhandled failure scenarios. While the backup 
implementation tries
-to correct all errors on its own, this tool may be necessary in the cases 
where the system cannot
-automatically recover on its own.
-$ hbase backup repair
-#### Positional Command-Line Arguments
-### Named Command-Line Arguments
-#### Example usage
-$ hbase backup repair
-## Configuration keys
-The backup and restore feature includes both required and optional 
configuration keys.
-### Required properties
-_hbase.backup.enable_: Controls whether or not the feature is enabled 
(Default: `false`). Set this value to `true`.
-_hbase.master.logcleaner.plugins_: A comma-separated list of classes invoked 
when cleaning logs in the HBase Master. Set
-this value to `org.apache.hadoop.hbase.backup.master.BackupLogCleaner` or 
append it to the current value.
-_hbase.procedure.master.classes_: A comma-separated list of classes invoked 
with the Procedure framework in the Master. Set
-this value to 
`org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager` or append 
it to the current value.
-_hbase.procedure.regionserver.classes_: A comma-separated list of classes 
invoked with the Procedure framework in the RegionServer.
-Set this value to 
 or append it to the current value.
-_hbase.coprocessor.region.classes_: A comma-separated list of RegionObservers 
deployed on tables. Set this value to
-`org.apache.hadoop.hbase.backup.BackupObserver` or append it to the current 
-_hbase.master.hfilecleaner.plugins_: A comma-separated list of HFileCleaners 
deployed on the Master. Set this value
-to `org.apache.hadoop.hbase.backup.BackupHFileCleaner` or append it to the 
current value.
-### Optional properties
-_hbase.backup.system.ttl_: The time-to-live in seconds of data in the 
`hbase:backup` tables (default: forever). This property
-is only relevant prior to the creation of the `hbase:backup` table. Use the 
`alter` command in the HBase shell to modify the TTL
-when this table already exists. See the <<br.filesystem.growth.warning,below 
section>> for more details on the impact of this
-configuration property.
-_hbase.backup.attempts.max_: The number of attempts to perform when taking 
hbase table snapshots (default: 10).
-_hbase.backup.attempts.pause.ms_: The amount of time to wait between failed 
snapshot attempts in milliseconds (default: 10000).
-_hbase.backup.logroll.timeout.millis_: The amount of time (in milliseconds) to 
wait for RegionServers to execute a WAL rolling
-in the Master's procedure framework (default: 30000).
-## Best Practices
-### Formulate a restore strategy and test it.
-Before you rely on a backup and restore strategy for your production 
environment, identify how backups must be performed,
-and more importantly, how restores must be performed. Test the plan to ensure 
that it is workable.
-At a minimum, store backup data from a production cluster on a different 
cluster or server. To further safeguard the data,
-use a backup location that is at a different physical location.
-If you have a unrecoverable loss of data on your primary production cluster as 
a result of computer system issues, you may
-be able to restore the data from a different cluster or server at the same 
site. However, a disaster that destroys the whole
-site renders locally stored backups useless. Consider storing the backup data 
and necessary resources (both computing capacity
-and operator expertise) to restore the data at a site sufficiently remote from 
the production site. In the case of a catastrophe
-at the whole primary site (fire, earthquake, etc.), the remote backup site can 
be very valuable.
-### Secure a full backup image first.
-As a baseline, you must complete a full backup of HBase data at least once 
before you can rely on incremental backups. The full
-backup should be stored outside of the source cluster. To ensure complete 
dataset recovery, you must run the restore utility
-with the option to restore baseline full backup. The full backup is the 
foundation of your dataset. Incremental backup data
-is applied on top of the full backup during the restore operation to return 
you to the point in time when backup was last taken.
-### Define and use backup sets for groups of tables that are logical subsets 
of the entire dataset.
-You can group tables into an object called a backup set. A backup set can save 
time when you have a particular group of tables
-that you expect to repeatedly back up or restore.
-When you create a backup set, you type table names to include in the group. 
The backup set includes not only groups of related
-tables, but also retains the HBase backup metadata. Afterwards, you can invoke 
the backup set name to indicate what tables apply
-to the command execution instead of entering all the table names individually.
-### Document the backup and restore strategy, and ideally log information 
about each backup.
-Document the whole process so that the knowledge base can transfer to new 
administrators after employee turnover. As an extra
-safety precaution, also log the calendar date, time, and other relevant 
details about the data of each backup. This metadata
-can potentially help locate a particular dataset in case of source cluster 
failure or primary site disaster. Maintain duplicate
-copies of all documentation: one copy at the production cluster site and 
another at the backup location or wherever it can be
-accessed by an administrator remotely from the production cluster.
-## Scenario: Safeguarding Application Datasets on Amazon S3
-This scenario describes how a hypothetical retail business uses backups to 
safeguard application data and then restore the dataset
-after failure.
-The HBase administration team uses backup sets to store data from a group of 
tables that have interrelated information for an
-application called green. In this example, one table contains transaction 
records and the other contains customer details. The
-two tables need to be backed up and be recoverable as a group.
-The admin team also wants to ensure daily backups occur automatically.
-.Tables Composing The Backup Set
-The following is an outline of the steps and examples of commands that are 
used to backup the data for the _green_ application and
-to recover the data later. All commands are run when logged in as HBase 
-1. A backup set called _green_set_ is created as an alias for both the 
transactions table and the customer table. The backup set can
-be used for all operations to avoid typing each table name. The backup set 
name is case-sensitive and should be formed with only
-printable characters and without spaces.
-$ hbase backup set add green_set transactions
-$ hbase backup set add green_set customer
-2. The first backup of green_set data must be a full backup. The following 
command example shows how credentials are passed to Amazon
-S3 and specifies the file system with the s3a: prefix.
-$ SECRET_KEY=123456789abcdefghijklmnopqrstuvwxyzABCD
-$ sudo -u hbase hbase backup create full\
-  s3a://$ACCESS_KEY:SECRET_KEY@prodhbasebackups/backups -s green_set
-3. Incremental backups should be run according to a schedule that ensures 
essential data recovery in the event of a catastrophe. At
-this retail company, the HBase admin team decides that automated daily backups 
secures the data sufficiently. The team decides that
-they can implement this by modifying an existing Cron job that is defined in 
`/etc/crontab`. Consequently, IT modifies the Cron job
-by adding the following line:
-@daily hbase hbase backup create incremental 
s3a://$ACCESS_KEY:$SECRET_KEY@prodhbasebackups/backups -s green_set
-4. A catastrophic IT incident disables the production cluster that the green 
application uses. An HBase system administrator of the
-backup cluster must restore the _green_set_ dataset to the point in time 
closest to the recovery objective.
-NOTE: If the administrator of the backup HBase cluster has the backup ID with 
relevant details in accessible records, the following
-search with the `hdfs dfs -ls` command and manually scanning the backup ID 
list can be bypassed. Consider continuously maintaining
-and protecting a detailed log of backup IDs outside the production cluster in 
your environment.
-The HBase administrator runs the following command on the directory where 
backups are stored to print the list of successful backup
-IDs on the console:
-`hdfs dfs -ls -t /prodhbasebackups/backups`
-5. The admin scans the list to see which backup was created at a date and time 
closest to the recovery objective. To do this, the
-admin converts the calendar timestamp of the recovery point in time to Unix 
time because backup IDs are uniquely identified with
-Unix time. The backup IDs are listed in reverse chronological order, meaning 
the most recent successful backup appears first.
-The admin notices that the following line in the command output corresponds 
with the _green_set_ backup that needs to be restored:
-6. The admin restores green_set invoking the backup ID and the -overwrite 
option. The -overwrite option truncates all existing data
-in the destination and populates the tables with data from the backup dataset. 
Without this flag, the backup data is appended to the
-existing data in the destination. In this case, the admin decides to overwrite 
the data because it is corrupted.
-$ sudo -u hbase hbase restore -s green_set \
-  s3a://$ACCESS_KEY:$SECRET_KEY@prodhbasebackups/backups backup_1467823988425 
\ -overwrite
-## Security of Backup Data
-With this feature which makes copying data to remote locations, it's important 
to take a moment to clearly state the procedural
-concerns that exist around data security. Like the HBase replication feature, 
backup and restore provides the constructs to automatically
-copy data from within a corporate boundary to some system outside of that 
boundary. It is imperative when storing sensitive data that with backup and 
restore, much
-less any feature which extracts data from HBase, the locations to which data 
is being sent has undergone a security audit to ensure
-that only authenticated users are allowed to access that data.
-For example, with the above example of backing up data to S3, it is of the 
utmost importance that the proper permissions are assigned
-to the S3 bucket to ensure that only a minimum set of authorized users are 
allowed to access this data. Because the data is no longer
-being accessed via HBase, and its authentication and authorization controls, 
we must ensure that the filesystem storing that data is
-providing a comparable level of security. This is a manual step which users 
*must* implement on their own.
-## Technical Details of Incremental Backup and Restore
-HBase incremental backups enable more efficient capture of HBase table images 
than previous attempts at serial backup and restore
-solutions, such as those that only used HBase Export and Import APIs. 
Incremental backups use Write Ahead Logs (WALs) to capture
-the data changes since the previous backup was created. A WAL roll (create new 
WALs) is executed across all RegionServers to track
-the WALs that need to be in the backup.
-After the incremental backup image is created, the source backup files usually 
are on same node as the data source. A process similar
-to the DistCp (distributed copy) tool is used to move the source backup files 
to the target file systems. When a table restore operation
-starts, a two-step process is initiated. First, the full backup is restored 
from the full backup image. Second, all WAL files from
-incremental backups between the last full backup and the incremental backup 
being restored are converted to HFiles, which the HBase
-Bulk Load utility automatically imports as restored data in the table.
-You can only restore on a live HBase cluster because the data must be 
redistributed to complete the restore operation successfully.
-## A Warning on File System Growth
-As a reminder, incremental backups are implemented via retaining the 
write-ahead logs which HBase primarily uses for data durability.
-Thus, to ensure that all data needing to be included in a backup is still 
available in the system, the HBase backup and restore feature
-retains all write-ahead logs since the last backup until the next incremental 
backup is executed.
-Like HBase Snapshots, this can have an expectedly large impact on the HDFS 
usage of HBase for high volume tables. Take care in enabling
-and using the backup and restore feature, specifically with a mind to removing 
backup sessions when they are not actively being used.
-The only automated, upper-bound on retained write-ahead logs for backup and 
restore is based on the TTL of the `hbase:backup` system table which,
-as of the time this document is written, is infinite (backup table entries are 
never automatically deleted). This requires that administrators
-perform backups on a schedule whose frequency is relative to the amount of 
available space on HDFS (e.g. less available HDFS space requires
-more aggressive backup merges and deletions). As a reminder, the TTL can be 
altered on the `hbase:backup` table using the `alter` command
-in the HBase shell. Modifying the configuration property 
`hbase.backup.system.ttl` in hbase-site.xml after the system table exists has 
no effect.
-## Capacity Planning
-When designing a distributed system deployment, it is critical that some basic 
mathmatical rigor is executed to ensure sufficient computational
-capacity is available given the data and software requirements of the system. 
For this feature, the availability of network capacity is the largest
-bottleneck when estimating the performance of some implementation of backup 
and restore. The second most costly function is the speed at which
-data can be read/written.
-### Full Backups
-To estimate the duration of a full backup, we have to understand the general 
actions which are invoked:
-* Write-ahead log roll on each RegionServer: ones to tens of seconds per 
RegionServer in parallel. Relative to the load on each RegionServer.
-* Take an HBase snapshot of the table(s): tens of seconds. Relative to the 
number of regions and files that comprise the table.
-* Export the snapshot to the destination: see below. Relative to the size of 
the data and the network bandwidth to the destination.
-To approximate how long the final step will take, we have to make some 
assumptions on hardware. Be aware that these will *not* be accurate for your
-system -- these are numbers that your or your administrator know for your 
system. Let's say the speed of reading data from HDFS on a single node is
-capped at 80MB/s (across all Mappers that run on that host), a modern network 
interface controller (NIC) supports 10Gb/s, the top-of-rack switch can
-handle 40Gb/s, and the WAN between your clusters is 10Gb/s. This means that 
you can only ship data to your remote at a speed of 1.25GB/s -- meaning
-that 16 nodes (`1.25 * 1024 / 80 = 16`) participating in the ExportSnapshot 
should be able to fully saturate the link between clusters. With more
-nodes in the cluster, we can still saturate the network but at a lesser impact 
on any one node which helps ensure local SLAs are made. If the size
-of the snapshot is 10TB, this would full backup would take in the ballpark of 
2.5 hours (`10 * 1024 / 1.25 / (60 * 60) = 2.23hrs`)
-As a general statement, it is very likely that the WAN bandwidth between your 
local cluster and the remote storage is the largest
-bottleneck to the speed of a full backup.
-When the concern is restricting the computational impact of backups to a 
"production system", the above formulas can be reused with the optional
-command-line arguments to `hbase backup create`: `-b`, `-w`, `-q`. The `-b` 
option defines the bandwidth at which each worker (Mapper) would
-write data. The `-w` argument limits the number of workers that would be 
spawned in the DistCp job. The `-q` allows the user to specify a YARN
-queue which can limit the specific nodes where the workers will be spawned -- 
this can quarantine the backup workers performing the copy to
-a set of non-critical nodes. Relating the `-b` and `-w` options to our earlier 
equations: `-b` would be used to restrict each node from reading
-data at the full 80MB/s and `-w` is used to limit the job from spawning 16 
worker tasks.
-### Incremental Backup
-Like we did for full backups, we have to understand the incremental backup 
process to approximate its runtime and cost.
-* Identify new write-ahead logs since last full or incremental backup: 
negligible. Apriori knowledge from the backup system table(s).
-* Read, filter, and write "minimized" HFiles equivalent to the WALs: dominated 
by the speed of writing data. Relative to write speed of HDFS.
-* DistCp the HFiles to the destination: <<br.export.snapshot.cost,see above>>.
-For the second step, the dominating cost of this operation would be the 
re-writing the data (under the assumption that a majority of the
-data in the WAL is preserved). In this case, we can assume an aggregate write 
speed of 30MB/s per node. Continuing our 16-node cluster example,
-this would require approximately 15 minutes to perform this step for 50GB of 
data (50 * 1024 / 60 / 60 = 14.2). The amount of time to start the
-DistCp MapReduce job would likely dominate the actual time taken to copy the 
data (50 / 1.25 = 40 seconds) and can be ignored.
-## Limitations of the Backup and Restore Utility
-*Serial backup operations*
-Backup operations cannot be run concurrently. An operation includes actions 
like create, delete, restore, and merge. Only one active backup session is 
supported. link:https://issues.apache.org/jira/browse/HBASE-16391[HBASE-16391]
-will introduce multiple-backup sessions support.
-*No means to cancel backups*
-Both backup and restore operations cannot be canceled. 
-The workaround to cancel a backup would be to kill the client-side backup 
command (`control-C`), ensure all relevant MapReduce jobs have exited, and then
-run the `hbase backup repair` command to ensure the system backup metadata is 
-*Backups can only be saved to a single location*
-Copying backup information to multiple locations is an exercise left to the 
user. link:https://issues.apache.org/jira/browse/HBASE-15476[HBASE-15476] will
-introduce the ability to specify multiple-backup destinations intrinsically.
-*HBase superuser access is required*
-Only an HBase superuser (e.g. hbase) is allowed to perform backup/restore, can 
pose a problem for shared HBase installations. Current mitigations would require
-coordination with system administrators to build and deploy a backup and 
restore strategy 
-*Backup restoration is an online operation*
-To perform a restore from a backup, it requires that the HBase cluster is 
online as a caveat of the current implementation 
-*Some operations may fail and require re-run*
-The HBase backup feature is primarily client driven. While there is the 
standard HBase retry logic built into the HBase Connection, persistent errors 
in executing operations
-may propagate back to the client (e.g. snapshot failure due to region splits). 
The backup implementation should be moved from client-side into the ProcedureV2 
-in the future which would provide additional robustness around 
transient/retryable failures. The `hbase backup repair` command is meant to 
correct states which the system
-cannot automatically detect and recover from.
-*Avoidance of declaration of public API*
-While the Java API to interact with this feature exists and its implementation 
is separated from an interface, insufficient rigor has been applied to 
determine if
-it is exactly what we intend to ship to users. As such, it is marked as for a 
`Private` audience with the expectation that, as users begin to try the 
feature, there
-will be modifications that would necessitate breaking compatibility 
-*Lack of global metrics for backup and restore*
-Individual backup and restore operations contain metrics about the amount of 
work the operation included, but there is no centralized location (e.g. the 
Master UI)
-which present information for consumption 

diff --git a/src/main/asciidoc/_chapters/community.adoc 
index d141dbf..1a4d977 100644
--- a/src/main/asciidoc/_chapters/community.adoc
+++ b/src/main/asciidoc/_chapters/community.adoc
@@ -40,24 +40,6 @@ When the feature is ready for commit, 3 +1s from committers 
will get your featur
 See link:http://search-hadoop.com/m/asM982C5FkS1[HBase, mail # dev - Thoughts
               about large feature dev branches]
-.Patch +1 Policy
-The below policy is something we put in place 09/2012.
-It is a suggested policy rather than a hard requirement.
-We want to try it first to see if it works before we cast it in stone.
-Apache HBase is made of 
-Components have one or more <<owner,OWNER>>s.
-See the 'Description' field on the 
 JIRA page for who the current owners are by component.
-Patches that fit within the scope of a single Apache HBase component require, 
at least, a +1 by one of the component's owners before commit.
-If owners are absent -- busy or otherwise -- two +1s by non-owners will 
-Patches that span components need at least two +1s before they can be 
committed, preferably +1s by owners of components touched by the x-component 
patch (TODO: This needs tightening up but I think fine for first pass).
-Any -1 on a patch by anyone vetoes a patch; it cannot be committed until the 
justification for the -1 is addressed.
 .How to set fix version in JIRA on issue resolve
@@ -85,19 +67,34 @@ We also are currently in violation of this basic tenet -- 
replication at least k
 == Community Roles
-.Component Owner/Lieutenant
+=== Release Managers
+Each maintained release branch has a release manager, who volunteers to 
coordinate new features and bug fixes are backported to that release.
+The release managers are 
+If you would like your feature or bug fix to be included in a given release, 
communicate with that release manager.
+If this list goes out of date or you can't reach the listed person, reach out 
to someone else on the list.
+NOTE: End-of-life releases are not included in this list.
+.Release Managers
+[cols="1,1", options="header"]
+| Release
+| Release Manager
+| 1.2
+| Sean Busbey
-Component owners are listed in the description field on this Apache HBase JIRA 
-The owners are listed in the 'Description' field rather than in the 'Component 
Lead' field because the latter only allows us list one individual whereas it is 
encouraged that components have multiple owners.
+| 1.3
+| Mikhail Antonov
-Owners or component lieutenants are volunteers who are (usually, but not 
necessarily) expert in their component domain and may have an agenda on how 
they think their Apache HBase component should evolve.
+| 1.4
+| Andrew Purtell
-. Owners will try and review patches that land within their component's scope.
-. If applicable, if an owner has an agenda, they will publish their goals or 
the design toward which they are driving their component
+| 2.0
+| Michael Stack
-If you would like to be volunteer as a component owner, just write the dev 
list and we'll sign you up.
-Owners do not need to be committers.
 == Commit Message format

diff --git a/src/main/asciidoc/_chapters/compression.adoc 
index 23ceeaf..8fc1c55 100644
--- a/src/main/asciidoc/_chapters/compression.adoc
+++ b/src/main/asciidoc/_chapters/compression.adoc
@@ -441,7 +441,7 @@ $ hbase org.apache.hadoop.hbase.util.LoadTestTool -write 
1:10:100 -num_keys 1000
-== Enable Data Block Encoding
+=== Enable Data Block Encoding
 Codecs are built into HBase so no extra configuration is needed.
 Codecs are enabled on a table by setting the `DATA_BLOCK_ENCODING` property.

diff --git a/src/main/asciidoc/_chapters/configuration.adoc 
index a941545..1f75855 100644
--- a/src/main/asciidoc/_chapters/configuration.adoc
+++ b/src/main/asciidoc/_chapters/configuration.adoc
@@ -92,28 +92,46 @@ This section lists required services and some required 
system configuration.
-[cols="1,1,4", options="header"]
+The following table summarizes the recommendation of the HBase community wrt 
deploying on various Java versions. An entry of "yes" is meant to indicate a 
base level of testing and willingness to help diagnose and address issues you 
might run into. Similarly, an entry of "no" or "Not Supported" generally means 
that should you run into an issue the community is likely to ask you to change 
the Java environment before proceeding to help. In some cases, specific 
guidance on limitations (e.g. wether compiling / unit tests work, specific 
operational issues, etc) will also be noted.
+.Long Term Support JDKs are recommended
+HBase recommends downstream users rely on JDK releases that are marked as Long 
Term Supported (LTS) either from the OpenJDK project or vendors. As of March 
2018 that means Java 8 is the only applicable version and that the next likely 
version to see testing will be Java 11 near Q3 2018.
+.Java support by release line
+[cols="1,1,1,1,1", options="header"]
 |HBase Version
 |JDK 7
 |JDK 8
+|JDK 9
+|JDK 10
 |link:http://search-hadoop.com/m/YGbbsPxZ723m3as[Not Supported]
+|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported]
+|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported]
+|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported]
+|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported]
+|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported]
+|link:https://issues.apache.org/jira/browse/HBASE-20264[Not Supported]
-NOTE: HBase will neither build nor compile with Java 6.
+NOTE: HBase will neither build nor run with Java 6.
 NOTE: You must set `JAVA_HOME` on each node of your cluster. _hbase-env.sh_ 
provides a handy mechanism to do this.
@@ -123,11 +141,7 @@ ssh::
   HBase uses the Secure Shell (ssh) command and utilities extensively to 
communicate between cluster nodes. Each server in the cluster must be running 
`ssh` so that the Hadoop and HBase daemons can be managed. You must be able to 
connect to all nodes via SSH, including the local node, from the Master as well 
as any backup Master, using a shared key rather than a password. You can see 
the basic methodology for such a set-up in Linux or Unix systems at 
"<<passwordless.ssh.quickstart>>". If your cluster nodes use OS X, see the 
 Setting up Remote Desktop and Enabling Self-Login] on the Hadoop wiki.
-  HBase uses the local hostname to self-report its IP address. Both forward 
and reverse DNS resolving must work in versions of HBase previous to 0.92.0. 
The link:https://github.com/sujee/hadoop-dns-checker[hadoop-dns-checker] tool 
can be used to verify DNS is working correctly on the cluster. The project 
`README` file provides detailed instructions on usage.
-Loopback IP::
-  Prior to hbase-0.96.0, HBase only used the IP address `` to refer 
to `localhost`, and this was not configurable.
-  See <<loopback.ip,Loopback IP>> for more details.
+  HBase uses the local hostname to self-report its IP address.
   The clocks on cluster nodes should be synchronized. A small amount of 
variation is acceptable, but larger amounts of skew can cause erratic and 
unexpected behavior. Time synchronization is one of the first things to check 
if you see unexplained problems in your cluster. It is recommended that you run 
a Network Time Protocol (NTP) service, or another time-synchronization 
mechanism on your cluster and that all nodes look to the same service for time 
synchronization. See the 
link:http://www.tldp.org/LDP/sag/html/basic-ntp-config.html[Basic NTP 
Configuration] at [citetitle]_The Linux Documentation Project (TLDP)_ to set up 
@@ -171,14 +185,14 @@ Linux Shell::
   All of the shell scripts that come with HBase rely on the 
link:http://www.gnu.org/software/bash[GNU Bash] shell.
-  Prior to HBase 0.96, running HBase on Microsoft Windows was limited only for 
testing purposes.
   Running production systems on Windows machines is not recommended.
 === link:https://hadoop.apache.org[Hadoop](((Hadoop)))
-The following table summarizes the versions of Hadoop supported with each 
version of HBase.
+The following table summarizes the versions of Hadoop supported with each 
version of HBase. Older versions not appearing in this table are considered 
unsupported and likely missing necessary features, while newer versions are 
untested but may be suitable.
 Based on the version of HBase, you should select the most appropriate version 
of Hadoop.
 You can use Apache Hadoop, or a vendor's distribution of Hadoop.
 No distinction is made here.
@@ -205,18 +219,15 @@ Use the following legend to interpret this table:
 [cols="1,1,1,1", options="header"]
 | | HBase-1.2.x | HBase-1.3.x | HBase-2.0.x
-|Hadoop-2.0.x-alpha | X | X | X
-|Hadoop-2.1.0-beta | X | X | X
-|Hadoop-2.2.0 | X  | X | X
-|Hadoop-2.3.x | X  | X | X
 |Hadoop-2.4.x | S | S | X
 |Hadoop-2.5.x | S | S | X
 |Hadoop-2.6.0 | X | X | X
 |Hadoop-2.6.1+ | S | S | S
 |Hadoop-2.7.0 | X | X | X
 |Hadoop-2.7.1+ | S | S | S
-|Hadoop-2.8.0 | X | X | X
-|Hadoop-2.8.1 | X | X | X
+|Hadoop-2.8.[0-1] | X | X | X
+|Hadoop-2.8.2+ | NT | NT | NT
+|Hadoop-2.9.0 | X | X | X
 |Hadoop-3.0.0 | NT | NT | NT
@@ -238,16 +249,10 @@ HBase on top of an HDFS Encryption Zone. Failure to do so 
will result in cluster
 data loss. This patch is present in Apache Hadoop releases 2.6.1+.
-.Hadoop 2.7.x
-Hadoop version 2.7.0 is not tested or supported as the Hadoop PMC has 
explicitly labeled that release as not being stable. (reference the 
link:https://s.apache.org/hadoop-2.7.0-announcement[announcement of Apache 
Hadoop 2.7.0].)
-.Hadoop 2.8.x
+.Hadoop 2.y.0 Releases
-Hadoop version 2.8.0 and 2.8.1 are not tested or supported as the Hadoop PMC 
has explicitly labeled that releases as not being stable. (reference the 
link:https://s.apache.org/hadoop-2.8.0-announcement[announcement of Apache 
Hadoop 2.8.0] and 
link:https://s.apache.org/hadoop-2.8.1-announcement[announcement of Apache 
Hadoop 2.8.1].)
+Starting around the time of Hadoop version 2.7.0, the Hadoop PMC got into the 
habit of calling out new minor releases on their major version 2 release line 
as not stable / production ready. As such, HBase expressly advises downstream 
users to avoid running on top of these releases. Note that additionally the 
2.8.1 was release was given the same caveat by the Hadoop PMC. For reference, 
see the release announcements for 
link:https://s.apache.org/hadoop-2.7.0-announcement[Apache Hadoop 2.7.0], 
link:https://s.apache.org/hadoop-2.8.0-announcement[Apache Hadoop 2.8.0], 
link:https://s.apache.org/hadoop-2.8.1-announcement[Apache Hadoop 2.8.1], and 
link:https://s.apache.org/hadoop-2.9.0-announcement[Apache Hadoop 2.9.0].
 .Replace the Hadoop Bundled With HBase!
@@ -294,9 +299,6 @@ See also 
<<casestudies.max.transfer.threads,casestudies.max.transfer.threads>> a
 === ZooKeeper Requirements
 ZooKeeper 3.4.x is required.
-HBase makes use of the `multi` functionality that is only available since 
Zookeeper 3.4.0. The `hbase.zookeeper.useMulti` configuration property defaults 
to `true`.
-Refer to link:https://issues.apache.org/jira/browse/HBASE-12241[HBASE-12241 
(The crash of regionServer when taking deadserver's replication queue breaks 
replication)] and 
link:https://issues.apache.org/jira/browse/HBASE-6775[HBASE-6775 (Use ZK.multi 
when available for HBASE-6710 0.92/0.94 compatibility fix)] for background.
-The property is deprecated and useMulti is always enabled in HBase 2.0.
 == HBase run modes: Standalone and Distributed
@@ -543,14 +545,13 @@ Usually this ensemble location is kept out in the 
_hbase-site.xml_ and is picked
 If you are configuring an IDE to run an HBase client, you should include the 
_conf/_ directory on your classpath so _hbase-site.xml_ settings can be found 
(or add _src/test/resources_ to pick up the hbase-site.xml used by tests).
-Minimally, an HBase client needs hbase-client module in its dependencies when 
connecting to a cluster:
+For Java applications using Maven, including the hbase-shaded-client module is 
the recommended dependency when connecting to a cluster:
-  <artifactId>hbase-client</artifactId>
-  <version>1.2.4</version>
+  <artifactId>hbase-shaded-client</artifactId>
+  <version>2.0.0</version>
@@ -785,7 +786,6 @@ For most usage patterns, you should use automatic splitting.
 See <<manual_region_splitting_decisions,manual region splitting decisions>> 
for more information about manual region splitting.
 Instead of allowing HBase to split your regions automatically, you can choose 
to manage the splitting yourself.
-This feature was added in HBase 0.90.0.
 Manually managing splits works if you know your keyspace well, otherwise let 
HBase figure where to split for you.
 Manual splitting can mitigate region creation and movement under load.
 It also makes it so region boundaries are known and invariant (if you disable 
region splitting). If you use manual splits, it is easier doing staggered, 
time-based major compactions to spread out your network IO load.
@@ -811,13 +811,12 @@ Otherwise, the cluster can be prone to compaction storms 
with a large number of
 It is important to understand that the data growth causes compaction storms 
and not the manual split decision.
 If the regions are split into too many large regions, you can increase the 
major compaction interval by configuring `HConstants.MAJOR_COMPACTION_PERIOD`.
-HBase 0.90 introduced `org.apache.hadoop.hbase.util.RegionSplitter`, which 
provides a network-IO-safe rolling split of all regions.
+The `org.apache.hadoop.hbase.util.RegionSplitter` utility also provides a 
network-IO-safe rolling split of all regions.
 ==== Managed Compactions
 By default, major compactions are scheduled to run once in a 7-day period.
-Prior to HBase 0.96.x, major compactions were scheduled to happen once per day 
by default.
 If you need to control exactly when and how often major compaction runs, you 
can disable managed major compactions.
 See the entry for `hbase.hregion.majorcompaction` in the 
<<compaction.parameters,compaction.parameters>> table for details.
@@ -937,8 +936,8 @@ To enable monitoring and management from remote systems, 
you need to set system
 See the 
 documentation] for more information.
 Historically, besides above port mentioned, JMX opens two additional random 
TCP listening ports, which could lead to port conflict problem. (See 
link:https://issues.apache.org/jira/browse/HBASE-10289[HBASE-10289] for details)
-As an alternative, You can use the coprocessor-based JMX implementation 
provided by HBase.
-To enable it in 0.99 or above, add below property in _hbase-site.xml_:
+As an alternative, you can use the coprocessor-based JMX implementation 
provided by HBase.
+To enable it, add below property in _hbase-site.xml_:
@@ -1033,8 +1032,8 @@ The corresponding properties for port configuration are 
 == Dynamic Configuration
-Since HBase 1.0.0, it is possible to change a subset of the configuration 
without requiring a server restart.
-In the HBase shell, there are new operators, `update_config` and 
`update_all_config` that will prompt a server or all servers to reload 
+It is possible to change a subset of the configuration without requiring a 
server restart.
+In the HBase shell, the operations `update_config` and `update_all_config` 
will prompt a server or all servers to reload configuration.
 Only a subset of all configurations can currently be changed in the running 
 Here are those configurations:

diff --git a/src/main/asciidoc/_chapters/datamodel.adoc 
index 3674566..ba4961a 100644
--- a/src/main/asciidoc/_chapters/datamodel.adoc
+++ b/src/main/asciidoc/_chapters/datamodel.adoc
@@ -343,6 +343,7 @@ In particular:
 Below we describe how the version dimension in HBase currently works.
 See link:https://issues.apache.org/jira/browse/HBASE-2406[HBASE-2406] for 
discussion of HBase versions. 
link:https://www.ngdata.com/bending-time-in-hbase/[Bending time in HBase] makes 
for a good read on the version, or time, dimension in HBase.
 It has more detail on versioning than is provided here.
 As of this writing, the limitation _Overwriting values at existing timestamps_ 
mentioned in the article no longer holds in HBase.
 This section is basically a synopsis of this article by Bruno Dumon.
@@ -503,8 +504,42 @@ Otherwise, a delete marker with a timestamp in the future 
is kept until the majo
 NOTE: This behavior represents a fix for an unexpected change that was 
introduced in HBase 0.94, and was fixed in 
 The change has been backported to HBase 0.94 and newer branches.
+=== Optional New Version and Delete behavior in HBase-2.0.0
+In `hbase-2.0.0`, the operator can specify an alternate version and
+delete treatment by setting the column descriptor property
+`NEW_VERSION_BEHAVIOR` to true (To set a property on a column family
+descriptor, you must first disable the table and then alter the
+column family descriptor; see <<cf.keep.deleted>> for an example
+of editing an attribute on a column family descriptor).
+The 'new version behavior', undoes the limitations listed below
+whereby a `Delete` ALWAYS overshadows a `Put` if at the same
+location -- i.e. same row, column family, qualifier and timestamp
+-- regardless of which arrived first. Version accounting is also
+changed as deleted versions are considered toward total version count.
+This is done to ensure results are not changed should a major
+compaction intercede. See `HBASE-15968` and linked issues for
+Running with this new configuration currently costs; we factor
+the Cell MVCC on every compare so we burn more CPU. The slow
+down will depend. In testing we've seen between 0% and 25%
+If replicating, it is advised that you run with the new
+serial replication feature (See `HBASE-9465`; the serial
+replication feature did NOT make it into `hbase-2.0.0` but
+should arrive in a subsequent hbase-2.x release) as now
+the order in which Mutations arrive is a factor.
 === Current Limitations
+The below limitations are addressed in hbase-2.0.0. See
+the section above, <<new.version.behavior>>.
 ==== Deletes mask Puts
 Deletes mask puts, even puts that happened after the delete was entered.

Reply via email to