re-replicating under replicated blocks should be more dynamic
-------------------------------------------------------------
Key: HDFS-2537
URL: https://issues.apache.org/jira/browse/HDFS-2537
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Nathan Roberts
When a node fails or is decommissioned, a large number of blocks become
under-replicated. Since re-replication work is distributed, the hope would be
that all blocks could be restored to their desired replication factor in very
short order. This doesn't happen though because the load the cluster is willing
to devote to this activity is mostly static (controlled by configuration
variables). Since it's mostly static, the rate has to be set conservatively to
avoid overloading the cluster with replication work.
This problem is especially noticeable when you have lots of small blocks. It
can take many hours to re-replicate the blocks that were on a node while the
cluster is mostly idle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
