liuyiyang created HDFS-12119:
--------------------------------
Summary: Inconsistent "Number of Under-Replicated Blocks" shown
on HDFS web UI and fsck report
Key: HDFS-12119
URL: https://issues.apache.org/jira/browse/HDFS-12119
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: liuyiyang
Sometimes the information "Number of Under-Replicated Blocks" shown on NameNode
web UI is inconsistent with the "Under-replicated blocks" information shown in
fsck report.
It's easy to reproduce such a case as follows:
1、In a cluster with DN1(rack0)、DN2(rack1) and DN3(rack2) which stores a lot of
blocks, the replication factor is set to 2;
2、Re-allocate racks as DN1(rack0)、DN2(rack1) and DN3(rack1) ;
3、Restart HDFS daemons.
Then you can find inconsistent "Number of Under-Replicated Blocks" on web ui
and "Under-replicated blocks" in fsck result.
I dug into the source code and found that "Number of Under-Replicated Blocks"
on web ui consists of blocks that have less than target number of replicas and
blocks that have the right number of replicas, but which the block manager felt
were badly distributed. In fsck result, "Under-replicated blocks" are the
blocks that have less than target number of replicas, and "Mis-replicated
blocks" are the blocks that are badly distributed. So the Under-Replicated
Blocks info on web UI and fsck result may be inconsistent.
Since Under-Replicated Blocks means higer missing risk for blocks, when threre
is no blocks that have less than target replicas but a lot of blocks that have
the right number of blocks but are badlly distributed, the "Number of
Under-Replicated Blocks" on web UI will be same as number of "Mis-replicated
blocks", which is misleading for users.
It would be clear to make "Number of Under-Replicated Blocks" on web UI be
consistent with "Under-replicated blocks" in fsck result.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
