[
https://issues.apache.org/jira/browse/HBASE-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912306#action_12912306
]
HBase Review Board commented on HBASE-2001:
-------------------------------------------
Message from: "Mingjie Lai" <[email protected]>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
The diff actually contains 2 seperate patches: HBase-2001 and the one for
(HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relies
on HBASE-2002 + HBASE-2321 which patches are still under review. I have to
include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard is so
powerful :) and it disallow my diff to be based on some unchecked in patch.
Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number
of lines are more than 7k. I turned back and forth, but still don't have a good
idea to create the patch in order to reduce the review pain. However right now
I'm putting the whole patch for all the 3 issues. Here the list of file which
are only related to coprocessor:
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
src/main/resources/hbase-default.xml
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
==========================
(Here is a brief description. Please find much more details at the
package-info.java in the diff. I also post the package-info.html to
https://issues.apache.org/jira/browse/HBASE-2001 as an attachment.)
Coprocessors are code that runs in-process on each region server. Regions
contain references to the coprocessor implementation classes associated with
them. Coprocessor classes will be loaded either from local jars on the region
server's classpath or via the HDFS classloader.
Multiple types of coprocessors are provided to provide sufficient flexibility
for potential use cases. Right now there are:
* Coprocessor: provides region lifecycle management hooks, e.g., region
open/close/split/flush/compact operations.
* RegionObserver: provides hook for monitor table operations from client side,
such as table get/put/scan/delete, etc.
* CommandTarget: provides on demand triggers for any arbitrary function
executed at a region. One use case is column aggregation at region server.
Coprocessor:
A coprocessor is required to implement Coprocessor interface so that
coprocessor framework can manage it internally.
Another design goal of this interface is to provide simple features for making
coprocessors useful, while exposing no more internal state or control actions
of the region server than necessary and not exposing them directly.
RegionObserver
If the coprocessor implements the RegionObserver interface it can observe and
mediate client actions on the region.
CommandTarget:
Coprocessor and RegionObserver provide certain hooks for injecting user code
running at each region. These code will be triggerd with existing HTable and
HBaseAdmin operations at the certain hook points.
Through CommandTarget and dynamic RPC protocol, you can define your own
interface communicated between client and region server, i.e., you can specify
new passed parameters and return types for a method. And the new CommandTarget
methods can be triggered by calling client side dynamic RPC functions --
HTable.exec(...).
Coprocess loading
A customized coprocessor can be loaded by two different ways, by configuration,
or by HTableDescriptor for a newly created table.
(Currently we don't really have an on demand coprocessor loading machanism for
opened regions. However it should be easy to create a dedicated CommandTarget
for coprocessor loading)
This addresses bug HBase-2001.
http://issues.apache.org/jira/browse/HBase-2001
Diffs
-----
src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81
src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java fbdec0b
src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263
src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838
src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b
src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java a4810a6
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/HMaster.java bccdc0e
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java fdef130
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f2e4e7c
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bb3b382
src/main/resources/hbase-default.xml 5452fd1
src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
PRE-CREATION
Diff: http://review.cloudera.org/r/876/diff
Testing
-------
Thanks,
Mingjie
> Coprocessors: Colocate user code with regions
> ---------------------------------------------
>
> Key: HBASE-2001
> URL: https://issues.apache.org/jira/browse/HBASE-2001
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Attachments: asm-transformations.pdf,
> HBASE-2001-RegionObserver-2.patch, HBASE-2001-RegionObserver.patch,
> HBASE-2001.patch.gz
>
>
> Support user code that runs run next to each region in table. As regions
> split and move, coprocessor code should automatically move also.
> Use classloader which looks on HDFS.
> Associate a list of classes to load with each table. Put this in HRI so it
> inherits from table but can be changed on a per region basis (so then those
> region specific changes can inherited by daughters).
> Not completely arbitrary code, should require implementation of an interface
> with callbacks for:
> * Open
> * Close
> * Split
> * Compact
> * (Multi)get and scanner next()
> * (Multi)put
> * (Multi)delete
> Add method to HRegionInterface for invoking coprocessor methods and
> retrieving results.
> Add methods in o.a.h.h.regionserver or subpackage which implement convenience
> functions for coprocessor methods and consistent/controlled access to
> internals: store access, threading, persistent and ephemeral state, scratch
> storage, etc.
> GitHub: http://github.com/apurtell/hbase-coprocessor
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
