[
https://issues.apache.org/jira/browse/HBASE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450625#comment-13450625
]
nkeywal commented on HBASE-5843:
--------------------------------
Some tests and analysis around Distributed Split
Scenario with a single machine, killing only a region server
- dfs.replication = 2
- local HD. The test failed with the ramDrive.
- Start with 2 DN and 2 RS. Create a table with 100 regions in the second one.
The first holds meta & root.
- Insert 10M rows, distributed on all regions. That creates 8 logs files of
60Mb each.
- start 3 more DN & RS.
- kill -9 the second RS (NOT the datanode).
- Wait for the regions to be available again.
0.94
~180s detection time (sometimes less, it's strange. But always more than 120s).
~25s split (two tasks of ~10s each per regionserver)
~30s assignment
0.96
~0s detection time (because it's a kill -9, so we have HBASE-5844. A hw failure
would bring the same result as 0.94).
~25s split (two tasks of ~10s each)
~30s assignment. Other tests seems to show that the actual time is taken on
replaying the edits.
=> No difference in time here. Except the detection time, that will not play a
role in a HW failure.
=> A split task in done in 10s. If you have a reasonable cluster, you can
expect this task to takes 10s in production.
=> Same should apply to replaying edits, if the regions are well redistributed
among the other machines.
=> Still, the assignment/replaying could be faster. Analysis to do, and JIRA to
create.
As of today, if the regionserver crashes, we should have, in production:
- 0.94: ~50s (30s detection + 10s split + 10s assignment)
- 0.96: ~20s (0s detection + 10s split + 10s assignment)
If it's a HW failure, we need to take into account that we've lost a datanode
as well.
> Improve HBase MTTR - Mean Time To Recover
> -----------------------------------------
>
> Key: HBASE-5843
> URL: https://issues.apache.org/jira/browse/HBASE-5843
> Project: HBase
> Issue Type: Umbrella
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
>
> A part of the approach is described here:
> https://docs.google.com/document/d/1z03xRoZrIJmg7jsWuyKYl6zNournF_7ZHzdi0qz_B4c/edit
> The ideal target is:
> - failure impact client applications only by an added delay to execute a
> query, whatever the failure.
> - this delay is always inferior to 1 second.
> We're not going to achieve that immediately...
> Priority will be given to the most frequent issues.
> Short term:
> - software crash
> - standard administrative tasks as stop/start of a cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
