[
https://issues.apache.org/jira/browse/HBASE-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006293#comment-14006293
]
Vladimir Rodionov commented on HBASE-10070:
-------------------------------------------
{quote}
In the present HBase architecture, it is hard, probably impossible, to satisfy
constraints like 99th percentile of the reads will be served under 10 ms
{quote}
I did some quick math. To blame bottom 1% on RS down event, means, that, on
average (with 1 min MTTR), we should see ~ 14-15 RS down events per cluster
per day (1440 minutes). I think it is way above what we have in a real life.
Just a quick math. I am not saying that this is not worth doing, but it will
not give us 10ms 99% (this is what I am pretty sure about). Just saying. This
is only for RS down type of failures. We all know that HBase cluster may
experience other types of temporary disabilities, which may affect read request
latency:
* blocked writes under heavy load (probably reads as well?) - not sure.
Solution : tune configuration and throttle incoming requests
* blocked reads due to blocked writes (no available handlers to serve incoming
requests). Solution: have different pools for write/reads or use priority on
RPC requests (new feature, correct?)
* excessive GC (sometimes) - Solution: off heap, off heap, off heap.
* something else, I forgot or not aware about?
but all of these should be and must avoided in a properly configured and tuned
cluster.
So, this is basically, to mitigate serious events (RS down) , not transient
ones. To improve read request latency distribution there is one classic
solution that works for sure - cache, cache, cache.
> HBase read high-availability using timeline-consistent region replicas
> ----------------------------------------------------------------------
>
> Key: HBASE-10070
> URL: https://issues.apache.org/jira/browse/HBASE-10070
> Project: HBase
> Issue Type: New Feature
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HighAvailabilityDesignforreadsApachedoc.pdf
>
>
> In the present HBase architecture, it is hard, probably impossible, to
> satisfy constraints like 99th percentile of the reads will be served under 10
> ms. One of the major factors that affects this is the MTTR for regions. There
> are three phases in the MTTR process - detection, assignment, and recovery.
> Of these, the detection is usually the longest and is presently in the order
> of 20-30 seconds. During this time, the clients would not be able to read the
> region data.
> However, some clients will be better served if regions will be available for
> reads during recovery for doing eventually consistent reads. This will help
> with satisfying low latency guarantees for some class of applications which
> can work with stale reads.
> For improving read availability, we propose a replicated read-only region
> serving design, also referred as secondary regions, or region shadows.
> Extending current model of a region being opened for reads and writes in a
> single region server, the region will be also opened for reading in region
> servers. The region server which hosts the region for reads and writes (as in
> current case) will be declared as PRIMARY, while 0 or more region servers
> might be hosting the region as SECONDARY. There may be more than one
> secondary (replica count > 2).
> Will attach a design doc shortly which contains most of the details and some
> thoughts about development approaches. Reviews are more than welcome.
> We also have a proof of concept patch, which includes the master and regions
> server side of changes. Client side changes will be coming soon as well.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
