[Coprocessors] Generic external process host
--------------------------------------------
Key: HBASE-4047
URL: https://issues.apache.org/jira/browse/HBASE-4047
Project: HBase
Issue Type: New Feature
Components: coprocessors
Reporter: Andrew Purtell
Where HBase coprocessors deviate substantially from the design (as I understand
it) of Google's BigTable coprocessors is we've reimagined it as a framework for
internal extension. In contrast BigTable coprocessors run as separate processes
colocated with tablet servers. The essential trade off is between performance,
flexibility and possibility, and the ability to control and enforce resource
usage.
Since the initial design of HBase coprocessors some additional considerations
are in play:
- Developing computational frameworks sitting directly on top of HBase hosted
in coprocessor(s);
- Introduction of the map reduce next generation (mrng) resource management
model, and the probability that limits will be enforced via cgroups at the OS
level after this is generally available, e.g. when RHEL 6 deployments are
common;
- The possibility of deployment of HBase onto mrng-enabled Hadoop clusters via
the mrng resource manager and a HBase-specific application controller.
Therefore we should consider developing a coprocessor that is a generic host
for another coprocessor, but one that forks a child process, loads the target
coprocessor into the child, establishes a bidirectional pipe and uses an
eventing model and umbilical protocol to provide for the coprocessor loaded
into the child the same semantics as if it was loaded internally to the parent,
and (eventually) use available resource management capabilities on the platform
-- perhaps via the mrng resource controller or directly with cgroups -- to
limit the child as desired by system administrators or the application designer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
