[
https://issues.apache.org/jira/browse/HBASE-10353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Avik Dey updated HBASE-10353:
-----------------------------
Description:
Increasingly HBase is being asked to operate in server environments where the
Java Virtual Machine is still working to take optimum advantage of the server
resources, for example those backed by solid-state storage, or deployed on
large memory systems. Many HBase users also struggle in environments with
demanding and tight request latency SLAs. The Java sandbox does not yet have
facilities for taking advantage of OS level features for high throughput low
latency I/O, nor efficient access to interconnects for high performance
computing. Our test results show that even using G1 GC, Java server processes
that must operate within tight latency bounds must keep their heaps small
relative to the kind of RAM we can pack into a server today.
We would like this issue to serve as an umbrella for brainstorming and
proposals for an incremental evolution of HBase, through use of existing plug
points and careful introduction of new ones, to a complete architecture for
plugging in alternative I/O engines. As a consequence we will be able to
explore improvements within and outside the JVM. Such an alternative engine
would be expected to provide lower latency and higher throughput than what is
possible within the constraints of today's implementation. Other JIRAs already
cover moving allocations for the I/O path off heap. Once we have I/O code paths
built on direct buffers that conveniently provides substrate for efficient
boundary crossings from Java into native code, we can open subtasks for
proposing and discussing where those boundaries can be drawn. We will also open
subtasks for proposing the shape and scope of the first experimental
alternative I/O engine.
This work should maintain API and wire compatibility as well as preserve data
format consistency with today’s HBase, so HBase users can take advantage of
this work without any need to change client code or massage existing data.
was:
Increasingly HBase is being asked to operate in server environments where the
Java Virtual Machine is still working to take optimum advantage of the server
resources, for example those backed by solid-state storage, or deployed on
large memory systems. Many HBase users also struggle in environments with
demanding and tight request latency SLAs. The Java sandbox does not yet have
facilities for taking advantage of OS level features for high throughput low
latency I/O, nor efficient access to interconnects for high performance
computing. Our test results show that even using G1 GC, Java server processes
that must operate within tight latency bounds must keep their heaps small
relative to the kind of RAM we can pack into a server today.
We would like this issue to serve as an umbrella for brainstorming and
proposals for an incremental evolution of HBase, through use of existing plug
points and careful introduction of new ones, to a complete architecture for
plugging in alternative I/O engines. As a consequence we will be able to
explore improvements within and outside the JVM. Such an alternative engine
would be expected to provide lower latency and higher throughput than what is
possible within the constraints of today's implementation. Other JIRAs already
cover moving allocations for the I/O path off heap. Once we have I/O code paths
built on direct buffers that conveniently provides substrate for efficient
boundary crossings from Java into native code, we can open subtasks for
proposing and discussing where those boundaries can be drawn. We will also open
subtasks for proposing the shape and scope of the first experimental
alternative I/O engine.
This work should maintain API and wire compatibility as well preserve data
format consistency with today’s HBase, so HBase users can take advantage of
this work without any need to change client code or massage existing data.
> Alternative I/O Engine
> ----------------------
>
> Key: HBASE-10353
> URL: https://issues.apache.org/jira/browse/HBASE-10353
> Project: HBase
> Issue Type: Umbrella
> Reporter: Avik Dey
> Assignee: Andrew Purtell
>
> Increasingly HBase is being asked to operate in server environments where the
> Java Virtual Machine is still working to take optimum advantage of the server
> resources, for example those backed by solid-state storage, or deployed on
> large memory systems. Many HBase users also struggle in environments with
> demanding and tight request latency SLAs. The Java sandbox does not yet have
> facilities for taking advantage of OS level features for high throughput low
> latency I/O, nor efficient access to interconnects for high performance
> computing. Our test results show that even using G1 GC, Java server processes
> that must operate within tight latency bounds must keep their heaps small
> relative to the kind of RAM we can pack into a server today.
>
> We would like this issue to serve as an umbrella for brainstorming and
> proposals for an incremental evolution of HBase, through use of existing plug
> points and careful introduction of new ones, to a complete architecture for
> plugging in alternative I/O engines. As a consequence we will be able to
> explore improvements within and outside the JVM. Such an alternative engine
> would be expected to provide lower latency and higher throughput than what is
> possible within the constraints of today's implementation. Other JIRAs
> already cover moving allocations for the I/O path off heap. Once we have I/O
> code paths built on direct buffers that conveniently provides substrate for
> efficient boundary crossings from Java into native code, we can open subtasks
> for proposing and discussing where those boundaries can be drawn. We will
> also open subtasks for proposing the shape and scope of the first
> experimental alternative I/O engine.
>
> This work should maintain API and wire compatibility as well as preserve data
> format consistency with today’s HBase, so HBase users can take advantage of
> this work without any need to change client code or massage existing data.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
