[
https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14317443#comment-14317443
]
Andrew Wang commented on HDFS-7411:
-----------------------------------
bq. When a node have many small blocks, say 10m, then would setting 100k blocks
per node be slower then the original node base algorithm?
Could you explain how you chose this value? Taking some sample numbers, a
really big Hadoop cluster might be 4000 nodes and 300 million blocks. This is
an average of 75k blocks per node. My experience with smaller, denser clusters
is that they top out at 500k blocks per node, because there are issues with
block report processing above that. 10m is 20x that already dense number.
Even so, it's not clear that it would be slower. The new algo does incremental
scans, so it'll be a lot faster toward the end of decom. Also with the old
code, doing full scans of 5 10m block DNs is going to be many seconds (maybe
even >1min) of pause each time, which is also certainly not what an admin wants.
bq. Agree. This is the same as my propose mentioned multiple times.
I think it's a bit different. Chris's proposal is to also enforce a # of nodes
limit in the new code, not to use the old code via a config toggle. The current
patch already does detection of the old config, so it could be tweaked a bit to
do this.
> Refactor and improve decommissioning logic into DecommissionManager
> -------------------------------------------------------------------
>
> Key: HDFS-7411
> URL: https://issues.apache.org/jira/browse/HDFS-7411
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.5.1
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: hdfs-7411.001.patch, hdfs-7411.002.patch,
> hdfs-7411.003.patch, hdfs-7411.004.patch, hdfs-7411.005.patch,
> hdfs-7411.006.patch, hdfs-7411.007.patch, hdfs-7411.008.patch,
> hdfs-7411.009.patch, hdfs-7411.010.patch
>
>
> Would be nice to split out decommission logic from DatanodeManager to
> DecommissionManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
