Xiaolin Ha created HBASE-26347:
----------------------------------
Summary: Support detect and exclude slow DNs in fan-out of WAL
Key: HBASE-26347
URL: https://issues.apache.org/jira/browse/HBASE-26347
Project: HBase
Issue Type: New Feature
Components: wal
Affects Versions: 2.0.0, 3.0.0-alpha-2
Reporter: Xiaolin Ha
Assignee: Xiaolin Ha
We all knows the WAL sync performance directly affects the RPC process time.
And we use self-designed FanOutOneBlockAsyncDFSOutput to sync WAL entries,
which connect straightly to all the block located DNs. But when even one DN of
the locations is slow, e.g. some disk hardware failures, the WAL syncs slow.
And what's more, the hardware failure detected by the lower layer HDFS system
is not so sensitive.
We can detect slow DNs by the ACK time of packets in
FanOutOneBlockAsyncDFSOutput, and exclude them when add new blocks after log
rolled(rolling log can also be triggered by slow syncs). And shows this info in
UI. We can also invalid these excluded DN cache after a duration, to aware the
recovery of those DNs.
But anymore, this idea can quickly reduce the influence of slow DNs, and
improve the service availability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
