[
https://issues.apache.org/jira/browse/HBASE-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15478105#comment-15478105
]
stack commented on HBASE-16583:
-------------------------------
The 'Availability' issue -- how our current server 'architecture' can lock up
if a single slow DN -- is reason enough to revisit core. But I am also stuck at
the moment unable to drive all CPU on a recent generation server. The
bottleneck is more than just our architecture [see interesting quote below] but
I'd be interested in any revamp that ups our resource utilization.
Agree with [~Apache9] that rewriting HBase core in seastar-style would be
'difficult' and require a discipline that might be, 'ahem', in short supply
around these parts.
[~Apache9] and [~phil84111] I like the suggestions and the call-out of
'natural' boundaries. Lets write them out and write out how we can do stuff
like: "....thread swtich is not always needed when we cross these points." Is
it too ideal having a high-level goal and then going there piecemeal? (Though I
hate our thread-per-handler model. To change that, it'd be a big change).
“The typical design of NoSQL data stores...consists of a JVM which runs on top
of Linux, utilizes the page cache, and uses complex memory allocation
strategies to “trick” the JVM garbage collector to avoid stop-the-world pauses.
Such a design suffers from sudden latency hiccups, expensive locking, and low
throughput due to low processor utilization.” from Scylladb Markitecture Page
> Staged Event-Driven Architecture
> --------------------------------
>
> Key: HBASE-16583
> URL: https://issues.apache.org/jira/browse/HBASE-16583
> Project: HBase
> Issue Type: Umbrella
> Reporter: Phil Yang
>
> Staged Event-Driven Architecture (SEDA) splits request-handling logic into
> several stages, each stage is executed in a thread pool and they are
> connected by queues.
> Currently, in region server we use a thread pool to handle requests from
> client. The number of handlers is configurable, reading and writing use
> different pools. The current architecture has two limitations:
> Performance:
> Different part of the handling path has different bottleneck. For example,
> accessing MemStore and cache mainly consumes CPU but accessing HDFS mainly
> consumes network/disk IO. If we use SEDA and split them into two different
> stages, we can use different numbers for two pools according to the
> CPU/disk/network performance case by case.
> Availability:
> HBASE-16388 described a scene that if the client use a thread pool and use
> blocking methods to access region servers, only one slow server may exhaust
> most of threads of the client. For HBase, we are the client and HDFS
> datanodes are the servers. A slow datanode may exhaust most of handlers. The
> best way to resolve this issue is make HDFS requests non-blocking, which is
> exactly what SEDA does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
