[
https://issues.apache.org/jira/browse/HDFS-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322120#comment-14322120
]
Frode Halvorsen commented on HDFS-7787:
---------------------------------------
And later, it seems to get better :) Now (after another paramater-tuning for
faster replication) it replicates 43.000 blocks / hour. And every block is one
that has zero live replicas in the cluster :) It actually seems that the
name-nodes needs time to calculate which blocks has higher priority. Now I only
need three more days before I can take down the data-node :)
As it turns out, it might just be parameteres that made me believe that it had
a bad prioritizing-algorithm :) Too bad a lot of the parameters I now have
changed is undocumented, but 'revealed' in different forum-postings...
A quick look at logs on the active name-node reveals that it actually only ask
the decommissioning node to replicate. No other nodes is contacted, thus it now
only replicates nodes with no live replicas. It might be my parameter-settings,
but it could actually have asked any of the other 5 datanodes to replicate the
blocks with one live replica... I'll try to add even more replication-requests
per heartbeat to see if it is able to make the other datanodes do any work as
well.
> Split QUEUE_HIGHEST_PRIORITY in UnderReplicatedBlocks to give more priority
> to blocks on nodes being decomissioned
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-7787
> URL: https://issues.apache.org/jira/browse/HDFS-7787
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.6.0
> Environment: 2 namenodes HA, 6 datanodes in two racks
> Reporter: Frode Halvorsen
> Labels: balance, hdfs, replication-performance
>
> Each file has a setting of 3 replicas. split on different racks.
> After a simulated crash of one rack (shutdown of all nodes, deleted
> data-directory an started nodes) and decommssion of one of the nodes in the
> orther rack the replication does not follow 'normal' rules...
> My cluster has appx 25 mill files, and the one node I now try to decommision
> has 9 millions underreplicated blocks, and 3,5 million blocks with 'no live
> replicas'. After a restart of the node, it starts to replicate both types of
> blocks, but after a while, it only repliates under-replicated blocks with
> other live copies. I would think that the 'normal' way to do this would be to
> make sure that all blocks this node keeps the only copy of, should be the
> first to be replicated/balanced ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
