http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/cp.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/cp.adoc b/src/main/asciidoc/_chapters/cp.adoc index 45944b4..5f50b68 100644 --- a/src/main/asciidoc/_chapters/cp.adoc +++ b/src/main/asciidoc/_chapters/cp.adoc @@ -27,254 +27,212 @@ :icons: font :experimental: -HBase Coprocessors are modeled after the Coprocessors which are part of Google's BigTable -(http://research.google.com/people/jeff/SOCC2010-keynote-slides.pdf pages 41-42.). + -Coprocessor is a framework that provides an easy way to run your custom code directly on -Region Server. -The information in this chapter is primarily sourced and heavily reused from: +HBase Coprocessors are modeled after Google BigTable's coprocessor implementation +(http://research.google.com/people/jeff/SOCC2010-keynote-slides.pdf pages 41-42.). + +The coprocessor framework provides mechanisms for running your custom code directly on +the RegionServers managing your data. Efforts are ongoing to bridge gaps between HBase's +implementation and BigTable's architecture. For more information see +link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047]. + +The information in this chapter is primarily sourced and heavily reused from the following +resources: . Mingjie Lai's blog post link:https://blogs.apache.org/hbase/entry/coprocessor_introduction[Coprocessor Introduction]. . Gaurav Bhardwaj's blog post link:http://www.3pillarglobal.com/insights/hbase-coprocessors[The How To Of HBase Coprocessors]. +[WARNING] +.Use Coprocessors At Your Own Risk +==== +Coprocessors are an advanced feature of HBase and are intended to be used by system +developers only. Because coprocessor code runs directly on the RegionServer and has +direct access to your data, they introduce the risk of data corruption, man-in-the-middle +attacks, or other malicious data access. Currently, there is no mechanism to prevent +data corruption by coprocessors, though work is underway on +link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047]. ++ +In addition, there is no resource isolation, so a well-intentioned but misbehaving +coprocessor can severely degrade cluster performance and stability. +==== +== Coprocessor Overview -== Coprocessor Framework - -When working with any data store (like RDBMS or HBase) you fetch the data (in case of RDBMS you -might use SQL query and in case of HBase you use either Get or Scan). To fetch only relevant data -you filter it (for RDBMS you put conditions in 'WHERE' predicate and in HBase you use -link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/Filter.html[Filter]). -After fetching the desired data, you perform your business computation on the data. -This scenario is close to ideal for "small data", where few thousand rows and a bunch of columns -are returned from the data store. Now imagine a scenario where there are billions of rows and -millions of columns and you want to perform some computation which requires all the data, like -calculating average or sum. Even if you are interested in just few columns, you still have to -fetch all the rows. There are a few drawbacks in this approach as described below: - -. In this approach the data transfer (from data store to client side) will become the bottleneck, -and the time required to complete the operation is limited by the rate at which data transfer -takes place. -. It's not always possible to hold so much data in memory and perform computation. -. Bandwidth is one of the most precious resources in any data center. Operations like this may -saturate your data centerâs bandwidth and will severely impact the performance of your cluster. -. Your client code is becoming thick as you are maintaining the code for calculating average or -summation on client side. Not a major drawback when talking of severe issues like -performance/bandwidth but still worth giving consideration. - -In a scenario like this it's better to move the computation (i.e. user's custom code) to the data -itself (Region Server). Coprocessor helps you achieve this but you can do more than that. -There is another advantage that your code runs in parallel (i.e. on all Regions). -To give an idea of Coprocessor's capabilities, different people give different analogies. -The three most famous analogies for Coprocessor are: -[[cp_analogies]] -Triggers and Stored Procedure:: This is the most common analogy for Coprocessor. Observer -Coprocessor is compared to triggers because like triggers they execute your custom code when -certain event occurs (like Get or Put etc.). Similarly Endpoints Coprocessor is compared to the -stored procedures and you can perform custom computation on data directly inside the region server. +In HBase, you fetch data using a `Get` or `Scan`, whereas in an RDBMS you use a SQL +query. In order to fetch only the relevant data, you filter it using a HBase +link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/Filter.html[Filter] +, whereas in an RDBMS you use a `WHERE` predicate. -MapReduce:: As in MapReduce you move the computation to the data in the same way. Coprocessor -executes your custom computation directly on Region Servers, i.e. where data resides. That's why -some people compare Coprocessor to a small MapReduce jobs. +After fetching the data, you perform computations on it. This paradigm works well +for "small data" with a few thousand rows and several columns. However, when you scale +to billions of rows and millions of columns, moving large amounts of data across your +network will create bottlenecks at the network layer, and the client needs to be powerful +enough and have enough memory to handle the large amounts of data and the computations. +In addition, the client code can grow large and complex. -AOP:: Some people compare it to _Aspect Oriented Programming_ (AOP). As in AOP, you apply advice -(on occurrence of specific event) by intercepting the request and then running some custom code -(probably cross-cutting concerns) and then forwarding the request on its path as if nothing -happened (or even return it back). Similarly in Coprocessor you have this facility of intercepting -the request and running custom code and then forwarding it on its path (or returning it). +In this scenario, coprocessors might make sense. You can put the business computation +code into a coprocessor which runs on the RegionServer, in the same location as the +data, and returns the result to the client. +This is only one scenario where using coprocessors can provide benefit. Following +are some analogies which may help to explain some of the benefits of coprocessors. -Although Coprocessor derives its roots from Google's Bigtable but it deviates from it largely in -its design. Currently there are efforts going on to bridge this gap. For more information see -link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047]. +[[cp_analogies]] +=== Coprocessor Analogies -In HBase, to implement a Coprocessor certain steps must be followed as described below: +Triggers and Stored Procedure:: + An Observer coprocessor is similar to a trigger in a RDBMS in that it executes + your code either before or after a specific event (such as a `Get` or `Put`) + occurs. An endpoint coprocessor is similar to a stored procedure in a RDBMS + because it allows you to perform custom computations on the data on the + RegionServer itself, rather than on the client. -. Either your class should extend one of the Coprocessor classes (like -// Below URL is more than 100 characters long. -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver] -) or it should implement Coprocessor interfaces (like -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/Coprocessor.html[Coprocessor], -// Below URL is more than 100 characters long. -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/CoprocessorService.html[CoprocessorService]). +MapReduce:: + MapReduce operates on the principle of moving the computation to the location of + the data. Coprocessors operate on the same principal. -. Load the Coprocessor: Currently there are two ways to load the Coprocessor. + -Static:: Loading from configuration -Dynammic:: Loading via 'hbase shell' or via Java code using HTableDescriptor class). + -For more details see <<cp_loading,Loading Coprocessors>>. +AOP:: + If you are familiar with Aspect Oriented Programming (AOP), you can think of a coprocessor + as applying advice by intercepting a request and then running some custom code, + before passing the request on to its final destination (or even changing the destination). -. Finally your client-side code to call the Coprocessor. This is the easiest step, as HBase -handles the Coprocessor transparently and you don't have to do much to call the Coprocessor. +=== Coprocessor Implementation Overview -The framework API is provided in the -// Below URL is more than 100 characters long. -link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/package-summary.html[coprocessor] -package. + -Coprocessors are not designed to be used by the end users but by developers. Coprocessors are -executed directly on region server; therefore a faulty/malicious code can bring your region server -down. Currently there is no mechanism to prevent this, but there are efforts going on for this. -For more, see link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047]. + -Two different types of Coprocessors are provided by the framework, based on their functionality. +. Either your class should extend one of the Coprocessor classes, such as +link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver], +or it should implement the link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/Coprocessor.html[Coprocessor] +or +link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/CoprocessorService.html[CoprocessorService] +interface. +. Load the coprocessor, either statically (from the configuration) or dynamically, +using HBase Shell. For more details see <<cp_loading,Loading Coprocessors>>. +. Call the coprocessor from your client-side code. HBase handles the coprocessor +trapsparently. + +The framework API is provided in the +link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/package-summary.html[coprocessor] +package. == Types of Coprocessors -Coprocessor can be broadly divided into two categories: Observer and Endpoint. - -=== Observer -Observer Coprocessor are easy to understand. People coming from RDBMS background can compare them -to the triggers available in relational databases. Folks coming from programming background can -visualize it like advice (before and after only) available in AOP (Aspect Oriented Programming). -See <<cp_analogies, Coprocessor Analogy>> + -Coprocessors allows you to hook your custom code in two places during the life cycle of an event. + -First is just _before_ the occurrence of the event (just like 'before' advice in AOP or triggers -like 'before update'). All methods providing this kind feature will start with the prefix `pre`. + -For example if you want your custom code to get executed just before the `Put` operation, you can -use the override the -// Below URL is more than 100 characters long. -link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#prePut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`prePut`] -method of -// Below URL is more than 100 characters long. -link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionCoprocessor]. -This method has following signature: -[source,java] ----- -public void prePut (final ObserverContext e, final Put put, final WALEdit edit,final Durability -durability) throws IOException; ----- +=== Observer Coprocessors -Secondly, the Observer Coprocessor also provides hooks for your code to get executed just _after_ -the occurrence of the event (similar to after advice in AOP terminology or 'after update' triggers -). The methods giving this functionality will start with the prefix `post`. For example, if you -want your code to be executed after the 'Put' operation, you should consider overriding -// Below URL is more than 100 characters long. -link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#postPut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`postPut`] -method of -// Below URL is more than 100 characters long. -link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionCoprocessor]: -[source,java] ----- -public void postPut(final ObserverContext e, final Put put, final WALEdit edit, final Durability -durability) throws IOException; ----- - -In short, the following conventions are generally followed: + -Override _preXXX()_ method if you want your code to be executed just before the occurrence of the -event. + -Override _postXXX()_ method if you want your code to be executed just after the occurrence of the -event. + +Observer coprocessors are triggered either before or after a specific event occurs. +Observers that happen before an event use methods that start with a `pre` prefix, +such as link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#prePut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`prePut`]. Observers that happen just after an event override methods that start +with a `post` prefix, such as link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#postPut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`postPut`]. -.Use Cases for Observer Coprocessors: -Few use cases of the Observer Coprocessor are: -. *Security*: Before performing any operation (like 'Get', 'Put') you can check for permission in -the 'preXXX' methods. +==== Use Cases for Observer Coprocessors +Security:: + Before performing a `Get` or `Put` operation, you can check for permission using + `preGet` or `prePut` methods. -. *Referential Integrity*: Unlike traditional RDBMS, HBase doesn't have the concept of referential -integrity (foreign key). Suppose for example you have a requirement that whenever you insert a -record in 'users' table, a corresponding entry should also be created in 'user_daily_attendance' -table. One way you could solve this is by using two 'Put' one for each table, this way you are -throwing the responsibility (of the referential integrity) to the user. A better way is to use -Coprocessor and overriding 'postPut' method in which you write the code to insert the record in -'user_daily_attendance' table. This way client code is more lean and clean. +Referential Integrity:: + HBase does not directly support the RDBMS concept of refential integrity, also known + as foreign keys. You can use a coprocessor to enforce such integrity. For instance, + if you have a business rule that every insert to the `users` table must be followed + by a corresponding entry in the `user_daily_attendance` table, you could implement + a coprocessor to use the `prePut` method on `user` to insert a record into `user_daily_attendance`. -. *Secondary Index*: Coprocessor can be used to maintain secondary indexes. For more information -see link:http://wiki.apache.org/hadoop/Hbase/SecondaryIndexing[SecondaryIndexing]. +Secondary Indexes:: + You can use a coprocessor to maintain secondary indexes. For more information, see + link:http://wiki.apache.org/hadoop/Hbase/SecondaryIndexing[SecondaryIndexing]. ==== Types of Observer Coprocessor -Observer Coprocessor comes in following flavors: - -. *RegionObserver*: This Coprocessor provides the facility to hook your code when the events on -region are triggered. Most common example include 'preGet' and 'postGet' for 'Get' operation and -'prePut' and 'postPut' for 'Put' operation. For exhaustive list of supported methods (events) see -// Below URL is more than 100 characters long. -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionObserver]. - -. *Region Server Observer*: Provides hook for the events related to the RegionServer, such as -stopping the RegionServer and performing operations before or after merges, commits, or rollbacks. -For more details please refer -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionServerObserver.html[RegionServerObserver]. - -. *Master Observer*: This observer provides hooks for DDL like operation, such as create, delete, -modify table. For entire list of available methods see -// Below URL is more than 100 characters long. -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/MasterObserver.html[MasterObserver]. - -. *WAL Observer*: Provides hooks for WAL (Write-Ahead-Log) related operation. It has only two -method 'preWALWrite()' and 'postWALWrite()'. For more details see -// Below URL is more than 100 characters long. -link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/WALObserver.html[WALObserver]. - -For example see <<cp_example,Examples>> +RegionObserver:: + A RegionObserver coprocessor allows you to observe events on a region, such as `Get` + and `Put` operations. See + link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionObserver]. + Consider overriding the convenience class + link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver], + which implements the `RegionObserver` interface and will not break if new methods are added. + +RegionServerObserver:: + A RegionServerObserver allows you to observe events related to the RegionServer's + operation, such as starting, stopping, or performing merges, commits, or rollbacks. + See + link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionServerObserver.html[RegionServerObserver]. + Consider overriding the convenience class + link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseMasterRegionServerObserver.html[BaseMasterRegionServerObserver] + which implements both `MasterObserver` and `RegionServerObserver` interfaces and + will not break if new methods are added. + +MasterOvserver:: + A MasterObserver allows you to observe events related to the HBase Master, such + as table creation, deletion, or schema modification. See + link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/MasterObserver.html[MasterObserver]. + Consider overriding the convenience class + link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseMasterRegionServerObserver.html[BaseMasterRegionServerObserver], + which implements both `MasterObserver` and `RegionServerObserver` interfaces and + will not break if new methods are added. + +WalObserver:: + A WalObserver allows you to observe events related to writes to the Write-Ahead + Log (WAL). See + link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/WALObserver.html[WALObserver]. + Consider overriding the convenience class + link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseWALObserver.html[BaseWALObserver], + which implements the `WalObserver` interface and will not break if new methods are added. + +<<cp_example,Examples>> provides working examples of observer coprocessors. === Endpoint Coprocessor -Endpoint Coprocessor can be compared to stored procedure found in RDBMS. -See <<cp_analogies, Coprocessor Analogy>>. They help in performing computation which is not -possible either through Observer Coprocessor or otherwise. For example, calculating average or -summation over the entire table that spans across multiple regions. They do so by providing a hook -for your custom code and then running it across all regions. + -With Endpoints Coprocessor you can create your own dynamic RPC protocol and thus can provide -communication between client and region server, hence enabling you to run your custom code on -region server (on each region of a table). + -Unlike observer Coprocessor (where your custom code is -executed transparently when events like 'Get' operation occurs), in Endpoint Coprocessor you have -to explicitly invoke the Coprocessor by using the -// Below URL is more than 100 characters long. +Endpoint processors allow you to perform computation at the location of the data. +See <<cp_analogies, Coprocessor Analogy>>. An example is the need to calculate a running +average or summation for an entire table which spans hundreds of regions. + +In contract to observer coprocessors, where your code is run transparently, endpoint +coprocessors must be explicitly invoked using the link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/Table.html#coprocessorService%28java.lang.Class,%20byte%5B%5D,%20byte%5B%5D,%20org.apache.hadoop.hbase.client.coprocessor.Batch.Call%29[CoprocessorService()] method available in -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/Table.html[Table] -(or -// Below URL is more than 100 characters long. -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTableInterface.html[HTableInterface] +link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/Table.html[Table], +link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTableInterface.html[HTableInterface], or -link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTable.html[HTable]). +link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTable.html[HTable]. -From version 0.96, implementing Endpoint Coprocessor is not straight forward. Now it is done with -the help of Google's Protocol Buffer. For more details on Protocol Buffer, please see +Starting with HBase 0.96, endpoint coprocessors are implemented using Google Protocol +Buffers (protobuf). For more details on protobuf, see Google's link:https://developers.google.com/protocol-buffers/docs/proto[Protocol Buffer Guide]. -Endpoints Coprocessor written in version 0.94 are not compatible with with version 0.96 or later -(for more details, see -link:https://issues.apache.org/jira/browse/HBASE-5448[HBASE-5448]), -so if your are upgrading your HBase cluster from version 0.94 (or before) to 0.96 (or later) you -have to rewrite your Endpoint coprocessor. - -For example see <<cp_example,Examples>> +Endpoints Coprocessor written in version 0.94 are not compatible with version 0.96 or later. +See +link:https://issues.apache.org/jira/browse/HBASE-5448[HBASE-5448]). To upgrade your +HBase cluster from 0.94 or earlier to 0.96 or later, you need to reimplement your +coprocessor. +<<cp_example,Examples>> provides working examples of endpoint coprocessors. [[cp_loading]] == Loading Coprocessors -_Loading of Coprocessor refers to the process of making your custom Coprocessor implementation -available to the the HBase, so that when a requests comes in or an event takes place the desired -functionality implemented in your custom code gets executed. + -Coprocessor can be loaded broadly in two ways. One is static (loading through configuration files) -and the other one is dynamic loading (using hbase shell or java code). +To make your coprocessor available to HBase, it must be _loaded_, either statically +(through the HBase configuration) or dynamically (using HBase Shell or the Java API). === Static Loading -Static loading means that your Coprocessor will take effect only when you restart your HBase and -there is a reason for it. In this you make changes 'hbase-site.xml' and therefore have to restart -HBase for your changes to take place. + -Following are the steps for loading Coprocessor statically. -. Define the Coprocessor in hbase-site.xml: Define a <property> element which consist of two -sub elements <name> and <value> respectively. +Follow these steps to statically load your coprocessor. Keep in mind that you must +restart HBase to unload a coprocessor that has been loaded statically. + +. Define the Coprocessor in _hbase-site.xml_, with a <property> element with a <name> +and a <value> sub-element. The <name> should be one of the following: + -.. <name> can have one of the following values: +- `hbase.coprocessor.region.classes` for RegionObservers and Endpoints. +- `hbase.coprocessor.wal.classes` for WALObservers. +- `hbase.coprocessor.master.classes` for MasterObservers. + -... 'hbase.coprocessor.region.classes' for RegionObservers and Endpoints. -... 'hbase.coprocessor.wal.classes' for WALObservers. -... 'hbase.coprocessor.master.classes' for MasterObservers. -.. <value> must contain the fully qualified class name of your class implmenting the Coprocessor. +<value> must contain the fully-qualified class name of your coprocessor's implementation +class. + For example to load a Coprocessor (implemented in class SumEndPoint.java) you have to create -following entry in RegionServer's 'hbase-site.xml' file (generally located under 'conf' directiory): +following entry in RegionServer's 'hbase-site.xml' file (generally located under 'conf' directory): + [source,xml] ---- @@ -283,6 +241,7 @@ following entry in RegionServer's 'hbase-site.xml' file (generally located under <value>org.myname.hbase.coprocessor.endpoint.SumEndPoint</value> </property> ---- ++ If multiple classes are specified for loading, the class names must be comma-separated. The framework attempts to load all the configured classes using the default class loader. Therefore, the jar file must reside on the server-side HBase classpath. @@ -297,34 +256,32 @@ When calling out to registered observers, the framework executes their callbacks sorted order of their priority. + Ties are broken arbitrarily. -. Put your code on classpth of HBase: There are various ways to do so, like adding jars on -classpath etc. One easy way to do this is to drop the jar (containing you code and all the -dependencies) in 'lib' folder of the HBase installation. - -. Restart the HBase. +. Put your code HBase's classpath. One easy way to do this is to drop the jar + (containing you code and all the dependencies) into the `lib/` directory in the + HBase installation. +. Restart HBase. -==== Unloading Static Coprocessor -Unloading static Coprocessor is easy. Following are the steps: -. Delete the Coprocessor's entry from the 'hbase-site.xml' i.e. remove the <property> tag. +=== Static Unloading -. Restart the Hbase. +. Delete the coprocessor's <property> element, including sub-elements, from `hbase-site.xml`. +. Restart HBase. +. Optionally, remove the coprocessor's JAR file from the classpath or HBase's `lib/` + directory. -. Optionally remove the Coprocessor jar file from the classpath (or from the lib directory if you -copied it over there). Removing the coprocessor JARs from HBaseâs classpath is a good practice. === Dynamic Loading -Dynamic loading refers to the process of loading Coprocessor without restarting HBase. This may -sound better than the static loading (and in some scenarios it may) but there is a caveat, dynamic -loaded Coprocessor applies to the table only for which it was loaded while same is not true for -static loading as it applies to all the tables. Due to this difference sometimes dynamically -loaded Coprocessor are also called *Table Coprocessor* (as they applies only to a single table) -while statically loaded Coprocessor are called *System Coprocessor* (as they applies to all the -tables). + -To dynamically load the Coprocessor you have to take the table offline hence during this time you -won't be able to process any request involving this table. + -There are three ways to dynamically load Coprocessor as shown below: + +You can also load a coprocessor dynamically, without restarting HBase. This may seem +preferable to static loading, but dynamically loaded coprocessors are loaded on a +per-table basis, and are only available to the table for which they were loaded. For +this reason, dynamically loaded tables are sometimes called *Table Coprocessor*. + +In addition, dynamically loading a coprocessor acts as a schema change on the table, +and the table must be taken offline to load the coprocessor. + +There are three ways to dynamically load Coprocessor. [NOTE] .Assumptions @@ -332,26 +289,25 @@ There are three ways to dynamically load Coprocessor as shown below: The below mentioned instructions makes the following assumptions: * A JAR called `coprocessor.jar` contains the Coprocessor implementation along with all of its -dependencies if any. +dependencies. * The JAR is available in HDFS in some location like `hdfs://<namenode>:<port>/user/<hadoop-user>/coprocessor.jar`. ==== -. *Using Shell*: You can load the Coprocessor using the HBase shell as follows: -.. Disable Table: Take table offline by disabling it. Suppose if the table name is 'users', then -to disable it enter following command: +==== Using HBase Shell + +. Disable the table using HBase Shell: + [source] ---- -hbase(main):001:0> disable 'users' +hbase> disable 'users' ---- -.. Load the Coprocessor: The Coprocessor jar should be on HDFS and should be accessible to HBase, -to load the Coprocessor use following command: +. Load the Coprocessor, using a command like the following: + [source] ---- -hbase(main):002:0> alter 'users', METHOD => 'table_att', 'Coprocessor'=>'hdfs://<namenode>:<port>/ +hbase alter 'users', METHOD => 'table_att', 'Coprocessor'=>'hdfs://<namenode>:<port>/ user/<hadoop-user>/coprocessor.jar| org.myname.hbase.Coprocessor.RegionObserverExample|1073741823| arg1=1,arg2=2' ---- @@ -370,30 +326,25 @@ observers registered at the same hook using priorities. This field can be left b case the framework will assign a default priority value. * Arguments (Optional): This field is passed to the Coprocessor implementation. This is optional. -.. Enable the table: To enable table type following command: +. Enable the table. + ---- hbase(main):003:0> enable 'users' ---- -.. Verification: This is optional but generally good practice to see if your Coprocessor is -loaded successfully. Enter following command: + +. Verify that the coprocessor loaded: + ---- hbase(main):04:0> describe 'users' ---- + -You must see some output like this: -+ ----- -DESCRIPTION ENABLED -'users', {TABLE_ATTRIBUTES => {coprocessor$1 => true 'hdfs://<namenode>:<port>/user/<hadoop-user>/ -coprocessor.jar| org.myname.hbase.Coprocessor.RegionObserverExample|1073741823|'}, {NAME => -'personalDet'..... ----- +The coprocessor should be listed in the `TABLE_ATTRIBUTES`. +==== Using the Java API (all HBase versions) + +The following Java code shows how to use the `setValue()` method of `HTableDescriptor` +to load a coprocessor on the `users` table. -. *Using setValue()* method of HTableDescriptor: This is done entirely in Java as follows: -+ [source,java] ---- TableName tableName = TableName.valueOf("users"); @@ -416,12 +367,14 @@ admin.modifyTable(tableName, hTableDescriptor); admin.enableTable(tableName); ---- -. *Using addCoprocessor()* method of HTableDescriptor: This method is available from 0.96 version -onwards. -+ +==== Using the Java API (HBase 0.96+ only) + +In HBase 0.96 and newer, the `addCoprocessor()` method of `HTableDescriptor` provides +an easier way to load a coprocessor dynamically. + [source,java] ---- -String tableName = "users"; +TableName tableName = TableName.valueOf("users"); String path = "hdfs://<namenode>:<port>/user/<hadoop-user>/coprocessor.jar"; Configuration conf = HBaseConfiguration.create(); HBaseAdmin admin = new HBaseAdmin(conf); @@ -439,26 +392,42 @@ admin.modifyTable(tableName, hTableDescriptor); admin.enableTable(tableName); ---- -==== WARNING: There is no guarantee that the framework will load a given Coprocessor successfully. For example, the shell command neither guarantees a jar file exists at a particular location nor verifies whether the given class is actually contained in the jar file. -==== -==== Unloading Dynamic Coprocessor -. Using shell: Run following command from HBase shell to remove Coprocessor from a table. +=== Dynamic Unloading + +==== Using HBase Shell + +. Disable the table. ++ +[source] +---- +hbase> disable 'users' +---- + +. Alter the table to remove the coprocessor. + [source] ---- -hbase(main):003:0> alter 'users', METHOD => 'table_att_unset', -hbase(main):004:0* NAME => 'coprocessor$1' +hbase> alter 'users', METHOD => 'table_att_unset', NAME => 'coprocessor$1' ---- -. Using HtableDescriptor: Simply reload the table definition _without_ setting the value of -Coprocessor either in setValue() or addCoprocessor() methods. This will remove the Coprocessor -attached to this table, if any. For example: +. Enable the table. + +[source] +---- +hbase> enable 'users' +---- + +==== Using the Java API + +Reload the table definition without setting the value of the coprocessor either by +using `setValue()` or `addCoprocessor()` methods. This will remove any coprocessor +attached to the table. + [source,java] ---- TableName tableName = TableName.valueOf("users"); @@ -477,26 +446,23 @@ hTableDescriptor.addFamily(columnFamily2); admin.modifyTable(tableName, hTableDescriptor); admin.enableTable(tableName); ---- -+ -Optionally you can also use removeCoprocessor() method of HTableDescriptor class. +In HBase 0.96 and newer, you can instead use the `removeCoprocessor()` method of the +`HTableDescriptor` class. [[cp_example]] == Examples -HBase ships Coprocessor examples for Observer Coprocessor see -// Below URL is more than 100 characters long. +HBase ships examples for Observer Coprocessor in link:http://hbase.apache.org/xref/org/apache/hadoop/hbase/coprocessor/example/ZooKeeperScanPolicyObserver.html[ZooKeeperScanPolicyObserver] -and for Endpoint Coprocessor see -// Below URL is more than 100 characters long. +and for Endpoint Coprocessor in link:http://hbase.apache.org/xref/org/apache/hadoop/hbase/coprocessor/example/RowCountEndpoint.html[RowCountEndpoint] A more detailed example is given below. -For the sake of example let's take an hypothetical case. Suppose there is a HBase table called -'users'. The table has two column families 'personalDet' and 'salaryDet' containing personal -details and salary details respectively. Below is the graphical representation of the 'users' -table. +These examples assume a table called `users`, which has two column families `personalDet` +and `salaryDet`, containing personal and salary details. Below is the graphical representation +of the `users` table. .Users Table [width="100%",cols="7",options="header,footer"] @@ -509,26 +475,22 @@ table. |==================== - === Observer Example -For the purpose of demonstration of Coprocessor we are assuming that 'admin' is a special person -and his details shouldn't be visible or returned to any client querying the 'users' table. + -To implement this functionality we will take the help of Observer Coprocessor. -Following are the implementation steps: + +The following Observer coprocessor prevents the details of the user `admin` from being +returned in a `Get` or `Scan` of the `users` table. . Write a class that extends the link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver] class. -. Override the 'preGetOp()' method (Note that 'preGet()' method is now deprecated). The reason for -overriding this method is to check if the client has queried for the rowkey with value 'admin' or -not. If the client has queried rowkey with 'admin' value then return the call without allowing the -system to perform the get operation thus saving on performance, otherwise process the request as -normal. +. Override the `preGetOp()` method (the `preGet()` method is deprecated) to check +whether the client has queried for the rowkey with value `admin`. If so, return an +empty result. Otherwise, process the request as normal. -. Put your code and dependencies in the jar file. +. Put your code and dependencies in a JAR file. -. Place the jar in HDFS where HBase can locate it. +. Place the JAR in HDFS where HBase can locate it. . Load the Coprocessor. @@ -536,8 +498,7 @@ normal. Following are the implementation of the above steps: -. For Step 1 and Step 2, below is the code. -+ + [source,java] ---- public class RegionObserverExample extends BaseRegionObserver { @@ -568,10 +529,10 @@ public class RegionObserverExample extends BaseRegionObserver { } } ---- -Overriding the 'preGetOp()' will only work for 'Get' operation. For 'Scan' operation it won't help -you. To deal with it you have to override another method called 'preScannerOpen()' method, and -add a Filter explicitly for admin as shown below: -+ + +Overriding the `preGetOp()` will only work for `Get` operations. You also need to override +the `preScannerOpen()` method to filter the `admin` row from scan results. + [source,java] ---- @Override @@ -583,12 +544,11 @@ final RegionScanner s) throws IOException { return s; } ---- -+ -This method works but there is a _side effect_. If the client has used any Filter in his scan, -then that Filter won't have any effect because our filter has replaced it. + -Another option you can try is to deliberately remove the admin from result. This approach is -shown below: -+ + +This method works but there is a _side effect_. If the client has used a filter in +its scan, that filter will be replaced by this filter. Instead, you can explicitly +remove any `admin` results from the scan: + [source,java] ---- @Override @@ -597,9 +557,9 @@ final List results, final int limit, final boolean hasMore) throws IOException { Result result = null; Iterator iterator = results.iterator(); while (iterator.hasNext()) { - result = iterator.next(); + result = iterator.next(); if (Bytes.equals(result.getRow(), ROWKEY)) { - iterator.remove(); + iterator.remove(); break; } } @@ -607,76 +567,12 @@ final List results, final int limit, final boolean hasMore) throws IOException { } ---- -. Step 3: It's pretty convenient to export the above program in a jar file. Let's assume that was -exported in a file called 'coprocessor.jar'. - -. Step 4: Copy the jar to HDFS. You may use command like this: -+ -[source] ----- -hadoop fs -copyFromLocal coprocessor.jar coprocessor.jar ----- - -. Step 5: Load the Coprocessor, see <<cp_loading,Loading of Coprocessor>>. - -. Step 6: Run the following program to test. The first part is testing 'Get' and second 'Scan'. -+ -[source,java] ----- -Configuration conf = HBaseConfiguration.create(); -// Use below code for HBase verion 1.x.x or above. -Connection connection = ConnectionFactory.createConnection(conf); -TableName tableName = TableName.valueOf("users"); -Table table = connection.getTable(tableName); - -//Use below code HBase verion 0.98.xx or below. -//HConnection connection = HConnectionManager.createConnection(conf); -//HTableInterface table = connection.getTable("users"); - -Get get = new Get(Bytes.toBytes("admin")); -Result result = table.get(get); -for (Cell c : result.rawCells()) { - System.out.println(Bytes.toString(CellUtil.cloneRow(c)) - + "==> " + Bytes.toString(CellUtil.cloneFamily(c)) - + "{" + Bytes.toString(CellUtil.cloneQualifier(c)) - + ":" + Bytes.toLong(CellUtil.cloneValue(c)) + "}"); -} -Scan scan = new Scan(); -ResultScanner scanner = table.getScanner(scan); -for (Result res : scanner) { - for (Cell c : res.rawCells()) { - System.out.println(Bytes.toString(CellUtil.cloneRow(c)) - + " ==> " + Bytes.toString(CellUtil.cloneFamily(c)) - + " {" + Bytes.toString(CellUtil.cloneQualifier(c)) - + ":" + Bytes.toLong(CellUtil.cloneValue(c)) - + "}"); - } -} ----- - === Endpoint Example -In our hypothetical example (See Users Table), to demonstrate the Endpoint Coprocessor we see a -trivial use case in which we will try to calculate the total (Sum) of gross salary of all -employees. One way of implementing Endpoint Coprocessor (for version 0.96 and above) is as follows: +Still using the `users` table, this example implements a coprocessor to calculate +the sum of all employee salaries, using an endpoint coprocessor. . Create a '.proto' file defining your service. - -. Execute the 'protoc' command to generate the Java code from the above '.proto' file. - -. Write a class that should: -.. Extend the above generated service class. -.. It should also implement two interfaces Coprocessor and CoprocessorService. -.. Override the service method. - -. Load the Coprocessor. - -. Write a client code to call Coprocessor. - -Implementation detail of the above steps is as follows: - -. Step 1: Create a 'proto' file to define your service, request and response. Let's call this file -"sum.proto". Below is the content of the 'sum.proto' file. + [source] ---- @@ -700,26 +596,25 @@ service SumService { } ---- -. Step 2: Compile the proto file using proto compiler (for detailed instructions see the -link:https://developers.google.com/protocol-buffers/docs/overview[official documentation]). +. Execute the `protoc` command to generate the Java code from the above .proto' file. + [source] ---- +$ mkdir src $ protoc --java_out=src ./sum.proto ---- + -[note] ----- -(Note: It is necessary for you to create the src folder). -This will generate a class call "Sum.java". ----- +This will generate a class call `Sum.java`. -. Step 3: Write your Endpoint Coprocessor: Firstly your class should extend the service just -defined above (i.e. Sum.SumService). Second it should implement Coprocessor and CoprocessorService -interfaces. Third, override the 'getService()', 'start()', 'stop()' and 'getSum()' methods. -Below is the full code: +. Write a class that extends the generated service class, implement the `Coprocessor` +and `CoprocessorService` classes, and override the service method. + -[source,java] +WARNING: If you load a coprocessor from `hbase-site.xml` and then load the same coprocessor +again using HBase Shell, it will be loaded a second time. The same class will +exist twice, and the second instance will have a higher ID (and thus a lower priority). +The effect is that the duplicate coprocessor is effectively ignored. ++ +[source, java] ---- public class SumEndPoint extends SumService implements Coprocessor, CoprocessorService { @@ -779,22 +674,16 @@ public class SumEndPoint extends SumService implements Coprocessor, CoprocessorS } } ---- - -. Step 4: Load the Coprocessor. See <<cp_loading,loading of Coprocessor>>. - -. Step 5: Now we have to write the client code to test it. To do so in your main method, write the -following code as shown below: + -[source,java] +[source, java] ---- - Configuration conf = HBaseConfiguration.create(); -// Use below code for HBase verion 1.x.x or above. +// Use below code for HBase version 1.x.x or above. Connection connection = ConnectionFactory.createConnection(conf); TableName tableName = TableName.valueOf("users"); Table table = connection.getTable(tableName); -//Use below code HBase verion 0.98.xx or below. +//Use below code HBase version 0.98.xx or below. //HConnection connection = HConnectionManager.createConnection(conf); //HTableInterface table = connection.getTable("users"); @@ -821,6 +710,86 @@ e.printStackTrace(); } ---- +. Load the Coprocessor. + +. Write a client code to call the Coprocessor. + + +== Guidelines For Deploying A Coprocessor + +Bundling Coprocessors:: + You can bundle all classes for a coprocessor into a + single JAR on the RegionServer's classpath, for easy deployment. Otherwise, + place all dependencies on the RegionServer's classpath so that they can be + loaded during RegionServer start-up. The classpath for a RegionServer is set + in the RegionServer's `hbase-env.sh` file. +Automating Deployment:: + You can use a tool such as Puppet, Chef, or + Ansible to ship the JAR for the coprocessor to the required location on your + RegionServers' filesystems and restart each RegionServer, to automate + coprocessor deployment. Details for such set-ups are out of scope of this + document. +Updating a Coprocessor:: + Deploying a new version of a given coprocessor is not as simple as disabling it, + replacing the JAR, and re-enabling the coprocessor. This is because you cannot + reload a class in a JVM unless you delete all the current references to it. + Since the current JVM has reference to the existing coprocessor, you must restart + the JVM, by restarting the RegionServer, in order to replace it. This behavior + is not expected to change. +Coprocessor Logging:: + The Coprocessor framework does not provide an API for logging beyond standard Java + logging. +Coprocessor Configuration:: + If you do not want to load coprocessors from the HBase Shell, you can add their configuration + properties to `hbase-site.xml`. In <<load_coprocessor_in_shell>>, two arguments are + set: `arg1=1,arg2=2`. These could have been added to `hbase-site.xml` as follows: +[source,xml] +---- +<property> + <name>arg1</name> + <value>1</value> +</property> +<property> + <name>arg2</name> + <value>2</value> +</property> +---- +Then you can read the configuration using code like the following: +[source,java] +---- +Configuration conf = HBaseConfiguration.create(); +// Use below code for HBase version 1.x.x or above. +Connection connection = ConnectionFactory.createConnection(conf); +TableName tableName = TableName.valueOf("users"); +Table table = connection.getTable(tableName); + +//Use below code HBase version 0.98.xx or below. +//HConnection connection = HConnectionManager.createConnection(conf); +//HTableInterface table = connection.getTable("users"); + +Get get = new Get(Bytes.toBytes("admin")); +Result result = table.get(get); +for (Cell c : result.rawCells()) { + System.out.println(Bytes.toString(CellUtil.cloneRow(c)) + + "==> " + Bytes.toString(CellUtil.cloneFamily(c)) + + "{" + Bytes.toString(CellUtil.cloneQualifier(c)) + + ":" + Bytes.toLong(CellUtil.cloneValue(c)) + "}"); +} +Scan scan = new Scan(); +ResultScanner scanner = table.getScanner(scan); +for (Result res : scanner) { + for (Cell c : res.rawCells()) { + System.out.println(Bytes.toString(CellUtil.cloneRow(c)) + + " ==> " + Bytes.toString(CellUtil.cloneFamily(c)) + + " {" + Bytes.toString(CellUtil.cloneQualifier(c)) + + ":" + Bytes.toLong(CellUtil.cloneValue(c)) + + "}"); + } +} +---- + + + == Monitor Time Spent in Coprocessors
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/datamodel.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/datamodel.adoc b/src/main/asciidoc/_chapters/datamodel.adoc index 7e9f0ab..66d2801 100644 --- a/src/main/asciidoc/_chapters/datamodel.adoc +++ b/src/main/asciidoc/_chapters/datamodel.adoc @@ -93,7 +93,7 @@ The colon character (`:`) delimits the column family from the column family _qua |=== |Row Key |Time Stamp |ColumnFamily `contents` |ColumnFamily `anchor`|ColumnFamily `people` |"com.cnn.www" |t9 | |anchor:cnnsi.com = "CNN" | -|"com.cnn.www" |t8 | |anchor:my.look.ca = "CNN.com" | +|"com.cnn.www" |t8 | |anchor:my.look.ca = "CNN.com" | |"com.cnn.www" |t6 | contents:html = "<html>..." | | |"com.cnn.www" |t5 | contents:html = "<html>..." | | |"com.cnn.www" |t3 | contents:html = "<html>..." | | @@ -171,7 +171,7 @@ For more information about the internals of how Apache HBase stores data, see << A namespace is a logical grouping of tables analogous to a database in relation database systems. This abstraction lays the groundwork for upcoming multi-tenancy related features: -* Quota Management (link:https://issues.apache.org/jira/browse/HBASE-8410[HBASE-8410]) - Restrict the amount of resources (ie regions, tables) a namespace can consume. +* Quota Management (link:https://issues.apache.org/jira/browse/HBASE-8410[HBASE-8410]) - Restrict the amount of resources (i.e. regions, tables) a namespace can consume. * Namespace Security Administration (link:https://issues.apache.org/jira/browse/HBASE-9206[HBASE-9206]) - Provide another level of security administration for tenants. * Region server groups (link:https://issues.apache.org/jira/browse/HBASE-6721[HBASE-6721]) - A namespace/table can be pinned onto a subset of RegionServers thus guaranteeing a course level of isolation. @@ -257,7 +257,7 @@ For example, the columns _courses:history_ and _courses:math_ are both members o The colon character (`:`) delimits the column family from the column family qualifier. The column family prefix must be composed of _printable_ characters. The qualifying tail, the column family _qualifier_, can be made of any arbitrary bytes. -Column families must be declared up front at schema definition time whereas columns do not need to be defined at schema time but can be conjured on the fly while the table is up an running. +Column families must be declared up front at schema definition time whereas columns do not need to be defined at schema time but can be conjured on the fly while the table is up and running. Physically, all column family members are stored together on the filesystem. Because tunings and storage specifications are done at the column family level, it is advised that all column family members have the same general access pattern and size characteristics. @@ -279,7 +279,7 @@ Gets are executed via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hba === Put -link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html[Put] either adds new rows to a table (if the key is new) or can update existing rows (if the key already exists). Puts are executed via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#put(org.apache.hadoop.hbase.client.Put)[Table.put] (writeBuffer) or link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#batch(java.util.List, java.lang.Object[])[Table.batch] (non-writeBuffer). +link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html[Put] either adds new rows to a table (if the key is new) or can update existing rows (if the key already exists). Puts are executed via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#put(org.apache.hadoop.hbase.client.Put)[Table.put] (writeBuffer) or link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#batch(java.util.List,%20java.lang.Object%5B%5D)[Table.batch] (non-writeBuffer). [[scan]] === Scans http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/developer.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc index 3cfc02f..d633569 100644 --- a/src/main/asciidoc/_chapters/developer.adoc +++ b/src/main/asciidoc/_chapters/developer.adoc @@ -90,7 +90,7 @@ We used to be on SVN. We migrated. See link:https://issues.apache.org/jira/browse/INFRA-7768[Migrate Apache HBase SVN Repos to Git]. See link:http://hbase.apache.org/source-repository.html[Source Code - Management] page for contributor and committer links or seach for HBase on the link:http://git.apache.org/[Apache Git] page. + Management] page for contributor and committer links or search for HBase on the link:http://git.apache.org/[Apache Git] page. == IDEs @@ -133,7 +133,7 @@ If you cloned the project via git, download and install the Git plugin (EGit). A ==== HBase Project Setup in Eclipse using `m2eclipse` The easiest way is to use the +m2eclipse+ plugin for Eclipse. -Eclipse Indigo or newer includes +m2eclipse+, or you can download it from link:http://www.eclipse.org/m2e//. It provides Maven integration for Eclipse, and even lets you use the direct Maven commands from within Eclipse to compile and test your project. +Eclipse Indigo or newer includes +m2eclipse+, or you can download it from http://www.eclipse.org/m2e/. It provides Maven integration for Eclipse, and even lets you use the direct Maven commands from within Eclipse to compile and test your project. To import the project, click and select the HBase root directory. `m2eclipse` locates all the hbase modules for you. @@ -146,7 +146,7 @@ If you install +m2eclipse+ and import HBase in your workspace, do the following ---- Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project hbase: -'An Ant BuildException has occured: Replace: source file .../target/classes/hbase-default.xml +'An Ant BuildException has occurred: Replace: source file .../target/classes/hbase-default.xml doesn't exist ---- + @@ -213,7 +213,7 @@ For additional information on setting up Eclipse for HBase development on Window === IntelliJ IDEA -You can set up IntelliJ IDEA for similar functinoality as Eclipse. +You can set up IntelliJ IDEA for similar functionality as Eclipse. Follow these steps. . Select @@ -227,7 +227,7 @@ Using the Eclipse Code Formatter plugin for IntelliJ IDEA, you can import the HB === Other IDEs -It would be userful to mirror the <<eclipse,eclipse>> set-up instructions for other IDEs. +It would be useful to mirror the <<eclipse,eclipse>> set-up instructions for other IDEs. If you would like to assist, please have a look at link:https://issues.apache.org/jira/browse/HBASE-11704[HBASE-11704]. [[build]] @@ -331,13 +331,13 @@ Tests may not all pass so you may need to pass `-DskipTests` unless you are incl ==== You will see ERRORs like the above title if you pass the _default_ profile; e.g. if you pass +hadoop.profile=1.1+ when building 0.96 or +hadoop.profile=2.0+ when building hadoop 0.98; just drop the hadoop.profile stipulation in this case to get your build to run again. -This seems to be a maven pecularity that is probably fixable but we've not spent the time trying to figure it. +This seems to be a maven peculiarity that is probably fixable but we've not spent the time trying to figure it. ==== Similarly, for 3.0, you would just replace the profile value. -Note that Hadoop-3.0.0-SNAPSHOT does not currently have a deployed maven artificat - you will need to build and install your own in your local maven repository if you want to run against this profile. +Note that Hadoop-3.0.0-SNAPSHOT does not currently have a deployed maven artifact - you will need to build and install your own in your local maven repository if you want to run against this profile. -In earilier versions of Apache HBase, you can build against older versions of Apache Hadoop, notably, Hadoop 0.22.x and 0.23.x. +In earlier versions of Apache HBase, you can build against older versions of Apache Hadoop, notably, Hadoop 0.22.x and 0.23.x. If you are running, for example HBase-0.94 and wanted to build against Hadoop 0.23.x, you would run with: [source,bourne] @@ -415,7 +415,7 @@ mvn -DskipTests package assembly:single deploy ==== Build Gotchas If you see `Unable to find resource 'VM_global_library.vm'`, ignore it. -Its not an error. +It's not an error. It is link:http://jira.codehaus.org/browse/MSITE-286[officially ugly] though. @@ -504,7 +504,7 @@ For building earlier versions, the process is different. See this section under the respective release documentation folders. .Point Releases -If you are making a point release (for example to quickly address a critical incompatability or security problem) off of a release branch instead of a development branch, the tagging instructions are slightly different. +If you are making a point release (for example to quickly address a critical incompatibility or security problem) off of a release branch instead of a development branch, the tagging instructions are slightly different. I'll prefix those special steps with _Point Release Only_. .Before You Begin @@ -516,7 +516,7 @@ You should also have tried recent branch tips out on a cluster under load, perha [NOTE] ==== At this point you should tag the previous release branch (ex: 0.96.1) with the new point release tag (e.g. -0.96.1.1 tag). Any commits with changes for the point release should be appled to the new tag. +0.96.1.1 tag). Any commits with changes for the point release should be applied to the new tag. ==== The Hadoop link:http://wiki.apache.org/hadoop/HowToRelease[How To @@ -584,8 +584,8 @@ $ mvn clean install -DskipTests assembly:single -Dassembly.file=hbase-assembly/s Extract the tarball and make sure it looks good. A good test for the src tarball being 'complete' is to see if you can build new tarballs from this source bundle. If the source tarball is good, save it off to a _version directory_, a directory somewhere where you are collecting all of the tarballs you will publish as part of the release candidate. -For example if you were building a hbase-0.96.0 release candidate, you might call the directory _hbase-0.96.0RC0_. -Later you will publish this directory as our release candidate up on http://people.apache.org/~YOU. +For example if you were building an hbase-0.96.0 release candidate, you might call the directory _hbase-0.96.0RC0_. +Later you will publish this directory as our release candidate up on pass:[http://people.apache.org/~YOU]. . Build the binary tarball. + @@ -1146,7 +1146,7 @@ However, maven will do this for us; just use: +mvn This is very similar to how you specify running a subset of unit tests (see above), but use the property `it.test` instead of `test`. To just run `IntegrationTestClassXYZ.java`, use: +mvn - failsafe:integration-test -Dit.test=IntegrationTestClassXYZ+ The next thing you might want to do is run groups of integration tests, say all integration tests that are named IntegrationTestClassX*.java: +mvn failsafe:integration-test -Dit.test=*ClassX*+ This runs everything that is an integration test that matches *ClassX*. This means anything matching: "**/IntegrationTest*ClassX*". You can also run multiple groups of integration tests using comma-delimited lists (similar to unit tests). Using a list of matches still supports full regex matching for each of the groups.This would look something like: +mvn + failsafe:integration-test -Dit.test=IntegrationTestClassXYZ+ The next thing you might want to do is run groups of integration tests, say all integration tests that are named IntegrationTestClassX*.java: +mvn failsafe:integration-test -Dit.test=*ClassX*+ This runs everything that is an integration test that matches *ClassX*. This means anything matching: "**/IntegrationTest*ClassX*". You can also run multiple groups of integration tests using comma-delimited lists (similar to unit tests). Using a list of matches still supports full regex matching for each of the groups. This would look something like: +mvn failsafe:integration-test -Dit.test=*ClassX*, *ClassY+ [[maven.build.commands.integration.tests.distributed]] @@ -1183,8 +1183,9 @@ For other deployment options, a ClusterManager can be implemented and plugged in [[maven.build.commands.integration.tests.destructive]] ==== Destructive integration / system tests (ChaosMonkey) -HBase 0.96 introduced a tool named `ChaosMonkey`, modeled after link:http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html -[same-named tool by Netflix's Chaos Monkey tool]. ChaosMonkey simulates real-world +HBase 0.96 introduced a tool named `ChaosMonkey`, modeled after +link:http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html[same-named tool by Netflix's Chaos Monkey tool]. +ChaosMonkey simulates real-world faults in a running cluster by killing or disconnecting random servers, or injecting other failures into the environment. You can use ChaosMonkey as a stand-alone tool to run a policy while other tests are running. In some environments, ChaosMonkey is @@ -1262,8 +1263,8 @@ HBase ships with several ChaosMonkey policies, available in the [[chaos.monkey.properties]] ==== Configuring Individual ChaosMonkey Actions -Since HBase version 1.0.0 (link:https://issues.apache.org/jira/browse/HBASE-11348 -[HBASE-11348]), ChaosMonkey integration tests can be configured per test run. +Since HBase version 1.0.0 (link:https://issues.apache.org/jira/browse/HBASE-11348[HBASE-11348]), +ChaosMonkey integration tests can be configured per test run. Create a Java properties file in the HBase classpath and pass it to ChaosMonkey using the `-monkeyProps` configuration flag. Configurable properties, along with their default values if applicable, are listed in the `org.apache.hadoop.hbase.chaos.factories.MonkeyConstants` @@ -1344,6 +1345,10 @@ NOTE: End-of-life releases are not included in this list. |=== | Release | Release Manager + +| 0.94 +| Lars Hofhansl + | 0.98 | Andrew Purtell @@ -1604,7 +1609,7 @@ All are subject to challenge of course but until then, please hold to the rules ZooKeeper state should transient (treat it like memory). If ZooKeeper state is deleted, hbase should be able to recover and essentially be in the same state. -* .ExceptionsThere are currently a few exceptions that we need to fix around whether a table is enabled or disabled. +* .Exceptions: There are currently a few exceptions that we need to fix around whether a table is enabled or disabled. * Replication data is currently stored only in ZooKeeper. Deleting ZooKeeper data related to replication may cause replication to be disabled. Do not delete the replication tree, _/hbase/replication/_. @@ -1866,9 +1871,9 @@ If the push fails for any reason, fix the problem or ask for help. Do not do a +git push --force+. + Before you can commit a patch, you need to determine how the patch was created. -The instructions and preferences around the way to create patches have changed, and there will be a transition periond. +The instructions and preferences around the way to create patches have changed, and there will be a transition period. + -* .Determine How a Patch Was CreatedIf the first few lines of the patch look like the headers of an email, with a From, Date, and Subject, it was created using +git format-patch+. +* .Determine How a Patch Was Created: If the first few lines of the patch look like the headers of an email, with a From, Date, and Subject, it was created using +git format-patch+. This is the preference, because you can reuse the submitter's commit message. If the commit message is not appropriate, you can still use the commit, then run the command +git rebase -i origin/master+, and squash and reword as appropriate. @@ -1971,7 +1976,7 @@ When the amending author is different from the original committer, add notice of from master to branch]. [[committer.tests]] -====== Committers are responsible for making sure commits do not break thebuild or tests +====== Committers are responsible for making sure commits do not break the build or tests If a committer commits a patch, it is their responsibility to make sure it passes the test suite. It is helpful if contributors keep an eye out that their patch does not break the hbase build and/or tests, but ultimately, a contributor cannot be expected to be aware of all the particular vagaries and interconnections that occur in a project like HBase. http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/external_apis.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/external_apis.adoc b/src/main/asciidoc/_chapters/external_apis.adoc index 44603f0..43a428a 100644 --- a/src/main/asciidoc/_chapters/external_apis.adoc +++ b/src/main/asciidoc/_chapters/external_apis.adoc @@ -77,7 +77,7 @@ of the <<security>> chapter. === Using REST Endpoints -The following examples use the placeholder server `http://example.com:8000`, and +The following examples use the placeholder server pass:[http://example.com:8000], and the following commands can all be run using `curl` or `wget` commands. You can request plain text (the default), XML , or JSON output by adding no header for plain text, or the header "Accept: text/xml" for XML or "Accept: application/json" for JSON. @@ -741,7 +741,7 @@ the data, and deletes the table. [source,jython] ---- import java.lang -from org.apache.hadoop.hbase import HBaseConfiguration, HTableDescriptor, HColumnDescriptor, HConstants +from org.apache.hadoop.hbase import HBaseConfiguration, HTableDescriptor, HColumnDescriptor, HConstants, TableName from org.apache.hadoop.hbase.client import HBaseAdmin, HTable, Get from org.apache.hadoop.hbase.io import Cell, RowResult @@ -753,7 +753,7 @@ conf = HBaseConfiguration() # Create a table named 'test' that has two column families, # one named 'content, and the other 'anchor'. The colons # are required for column family names. -tablename = "test" +tablename = TableName.valueOf("test") desc = HTableDescriptor(tablename) desc.addFamily(HColumnDescriptor("content:")) http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/faq.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/faq.adoc b/src/main/asciidoc/_chapters/faq.adoc index 22e4ad3..a622650 100644 --- a/src/main/asciidoc/_chapters/faq.adoc +++ b/src/main/asciidoc/_chapters/faq.adoc @@ -46,7 +46,7 @@ What is the history of HBase?:: === Upgrading How do I upgrade Maven-managed projects from HBase 0.94 to HBase 0.96+?:: - In HBase 0.96, the project moved to a modular structure. Adjust your project's dependencies to rely upon the `hbase-client` module or another module as appropriate, rather than a single JAR. You can model your Maven depency after one of the following, depending on your targeted version of HBase. See Section 3.5, âUpgrading from 0.94.x to 0.96.xâ or Section 3.3, âUpgrading from 0.96.x to 0.98.xâ for more information. + In HBase 0.96, the project moved to a modular structure. Adjust your project's dependencies to rely upon the `hbase-client` module or another module as appropriate, rather than a single JAR. You can model your Maven dependency after one of the following, depending on your targeted version of HBase. See Section 3.5, âUpgrading from 0.94.x to 0.96.xâ or Section 3.3, âUpgrading from 0.96.x to 0.98.xâ for more information. + .Maven Dependency for HBase 0.98 [source,xml] @@ -55,18 +55,18 @@ How do I upgrade Maven-managed projects from HBase 0.94 to HBase 0.96+?:: <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>0.98.5-hadoop2</version> -</dependency> ----- -+ -.Maven Dependency for HBase 0.96 +</dependency> +---- ++ +.Maven Dependency for HBase 0.96 [source,xml] ---- <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>0.96.2-hadoop2</version> -</dependency> ----- +</dependency> +---- + .Maven Dependency for HBase 0.94 [source,xml] @@ -75,9 +75,9 @@ How do I upgrade Maven-managed projects from HBase 0.94 to HBase 0.96+?:: <groupId>org.apache.hbase</groupId> <artifactId>hbase</artifactId> <version>0.94.3</version> -</dependency> ----- - +</dependency> +---- + === Architecture How does HBase handle Region-RegionServer assignment and locality?:: @@ -91,7 +91,7 @@ Where can I learn about the rest of the configuration options?:: See <<configuration>>. === Schema Design / Data Access - + How should I design my schema in HBase?:: See <<datamodel>> and <<schema>>. http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/getting_started.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/getting_started.adoc b/src/main/asciidoc/_chapters/getting_started.adoc index 4134831..1b38e6e 100644 --- a/src/main/asciidoc/_chapters/getting_started.adoc +++ b/src/main/asciidoc/_chapters/getting_started.adoc @@ -57,7 +57,7 @@ Prior to HBase 0.94.x, HBase expected the loopback IP address to be 127.0.0.1. U .Example /etc/hosts File for Ubuntu ==== -The following _/etc/hosts_ file works correctly for HBase 0.94.x and earlier, on Ubuntu. Use this as a template if you run into trouble. +The following _/etc/hosts_ file works correctly for HBase 0.94.x and earlier, on Ubuntu. Use this as a template if you run into trouble. [listing] ---- 127.0.0.1 localhost
