Stephen O'Donnell created HDFS-14854:
----------------------------------------
Summary: Create improved decommission monitor implementation
Key: HDFS-14854
URL: https://issues.apache.org/jira/browse/HDFS-14854
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.3.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
In HDFS-13157, we discovered a series of problems with the current decommission
monitor implementation, such as:
* Blocks are replicated sequentially disk by disk and node by node, and hence
the load is not spread well across the cluster
* Adding a node for decommission can cause the namenode write lock to be held
for a long time.
* Decommissioning nodes floods the replication queue and under replicated
blocks from a future node or disk failure may way for a long time before they
are replicated.
* Blocks pending replication are checked many times under a write lock before
they are sufficiently replicate, wasting resources
In this Jira I propose to create a new implementation of the decommission
monitor that resolves these issues. As it will be difficult to prove one
implementation is better than another, the new implementation can be enabled or
disabled giving the option of the existing implementation or the new one.
I will attach a pdf with some more details on the design and then a version 1
patch shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
