Avik Dey created HBASE-10353:
--------------------------------
Summary: Alternative I/O Engine
Key: HBASE-10353
URL: https://issues.apache.org/jira/browse/HBASE-10353
Project: HBase
Issue Type: Umbrella
Reporter: Avik Dey
Increasingly HBase is being asked to operate in server environments where the
Java Virtual Machine is still working to take optimum advantage of the server
resources, for example those backed by solid-state storage, or deployed on
large memory systems. Many HBase users also struggle in environments with
demanding and tight request latency SLAs. The Java sandbox does not yet have
facilities for taking advantage of OS level features for high throughput low
latency I/O, nor efficient access to interconnects for high performance
computing. Our test results show that even using G1 GC, Java server processes
that must operate within tight latency bounds must keep their heaps small
relative to the kind of RAM we can pack into a server today.
We would like this issue to serve as an umbrella for brainstorming and
proposals for an incremental evolution of HBase, through use of existing plug
points and careful introduction of new ones, to a complete architecture for
plugging in alternative I/O engines. As a consequence we will be able to
explore improvements within and outside the JVM. Such an alternative engine
would be expected to provide lower latency and higher throughput than what is
possible within the constraints of today's implementation. Other JIRAs already
cover moving allocations for the I/O path off heap. Once we have I/O code paths
built on direct buffers that conveniently provides substrate for efficient
boundary crossings from Java into native code, we can open subtasks for
proposing and discussing where those boundaries can be drawn. We will also open
subtasks for proposing the shape and scope of the first experimental
alternative I/O engine.
This work should maintain API and wire compatibility as well preserve data
format consistency with today’s HBase, so HBase users can take advantage of
this work without any need to change client code or massage existing data.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
