[jira] [Commented] (HDFS-16663) Allow block reconstruction pending timeout refreshable to increase decommission performance
[ https://issues.apache.org/jira/browse/HDFS-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570570#comment-17570570 ] Hiroyuki Adachi commented on HDFS-16663: [~caozhiqiang] I'm +1 (non-binding). > Allow block reconstruction pending timeout refreshable to increase > decommission performance > --- > > Key: HDFS-16663 > URL: https://issues.apache.org/jira/browse/HDFS-16663 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ec, namenode >Affects Versions: 3.4.0 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In HDFS-16613, increase the value of > dfs.namenode.replication.max-streams-hard-limit would maximize the IO > performance of the decommissioning DN, which has a lot of EC blocks. Besides > this, we also need to decrease the value of > dfs.namenode.reconstruction.pending.timeout-sec, default is 5 minutes, to > shorten the interval time for checking pendingReconstructions. Or the > decommissioning node would be idle to wait for copy tasks in most of this 5 > minutes. > In decommission progress, we may need to reconfigure these 2 parameters > several times. In HDFS-14560, the > dfs.namenode.replication.max-streams-hard-limit can already be reconfigured > dynamically without namenode restart. And the > dfs.namenode.reconstruction.pending.timeout-sec parameter also need to be > reconfigured dynamically. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16613) EC: Improve performance of decommissioning dn with many ec blocks
[ https://issues.apache.org/jira/browse/HDFS-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552722#comment-17552722 ] Hiroyuki Adachi edited comment on HDFS-16613 at 6/10/22 2:28 PM: - [~caozhiqiang] , thank you for explaining in detail. I think the data process you described is correct and your approach for improving performance is right. My concern was the reconstruction load on a large cluster where blocksToProcess is much larger than maxTransfers. But I found that I had misunderstood that the blocks held by the busy node would be reconstructed rather than replicated. So I think there is no problem using dfs.namenode.replication.max-streams-hard-limit for this purpose. Basically, it is for the highest priority replication, but the patch will not affect that since computeReconstructionWorkForBlocks() processes higher priority replication queue first, the decommissioning node will not fill up by low redundancy EC block replication tasks. was (Author: hadachi): [~caozhiqiang] , thank you for explaining in detail. I think the data process you described is correct and your approach for improving performance is right. My concern was the reconstruction load on a large cluster where blocksToProcess is much larger than maxTransfers. But I found that I had misunderstood that the blocks held by the busy node would be reconstructed rather than replicated. So I think there is no problem using dfs.namenode.replication.max-streams-hard-limit for this purpose. > EC: Improve performance of decommissioning dn with many ec blocks > - > > Key: HDFS-16613 > URL: https://issues.apache.org/jira/browse/HDFS-16613 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ec, erasure-coding, namenode >Affects Versions: 3.4.0 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Labels: pull-request-available > Attachments: image-2022-06-07-11-46-42-389.png, > image-2022-06-07-17-42-16-075.png, image-2022-06-07-17-45-45-316.png, > image-2022-06-07-17-51-04-876.png, image-2022-06-07-17-55-40-203.png, > image-2022-06-08-11-38-29-664.png, image-2022-06-08-11-41-11-127.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > In a hdfs cluster with a lot of EC blocks, decommission a dn is very slow. > The reason is unlike replication blocks can be replicated from any dn which > has the same block replication, the ec block have to be replicated from the > decommissioning dn. > The configurations dfs.namenode.replication.max-streams and > dfs.namenode.replication.max-streams-hard-limit will limit the replication > speed, but increase these configurations will create risk to the whole > cluster's network. So it should add a new configuration to limit the > decommissioning dn, distinguished from the cluster wide max-streams limit. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16613) EC: Improve performance of decommissioning dn with many ec blocks
[ https://issues.apache.org/jira/browse/HDFS-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552722#comment-17552722 ] Hiroyuki Adachi commented on HDFS-16613: [~caozhiqiang] , thank you for explaining in detail. I think the data process you described is correct and your approach for improving performance is right. My concern was the reconstruction load on a large cluster where blocksToProcess is much larger than maxTransfers. But I found that I had misunderstood that the blocks held by the busy node would be reconstructed rather than replicated. So I think there is no problem using dfs.namenode.replication.max-streams-hard-limit for this purpose. > EC: Improve performance of decommissioning dn with many ec blocks > - > > Key: HDFS-16613 > URL: https://issues.apache.org/jira/browse/HDFS-16613 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ec, erasure-coding, namenode >Affects Versions: 3.4.0 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Labels: pull-request-available > Attachments: image-2022-06-07-11-46-42-389.png, > image-2022-06-07-17-42-16-075.png, image-2022-06-07-17-45-45-316.png, > image-2022-06-07-17-51-04-876.png, image-2022-06-07-17-55-40-203.png, > image-2022-06-08-11-38-29-664.png, image-2022-06-08-11-41-11-127.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > In a hdfs cluster with a lot of EC blocks, decommission a dn is very slow. > The reason is unlike replication blocks can be replicated from any dn which > has the same block replication, the ec block have to be replicated from the > decommissioning dn. > The configurations dfs.namenode.replication.max-streams and > dfs.namenode.replication.max-streams-hard-limit will limit the replication > speed, but increase these configurations will create risk to the whole > cluster's network. So it should add a new configuration to limit the > decommissioning dn, distinguished from the cluster wide max-streams limit. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16613) EC: Improve performance of decommissioning dn with many ec blocks
[
https://issues.apache.org/jira/browse/HDFS-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550940#comment-17550940
]
Hiroyuki Adachi commented on HDFS-16613:
[~caozhiqiang] , thank you for your explanation. It looks good.
Now I understand that the blocksToProcess controls the number of replication
works, so if it is less than dfs.namenode.replication.max-streams-hard-limit,
all blocks use replication on decommissioning node but not reconstruction.
Could you please tell me the value of
dfs.namenode.replication.max-streams-hard-limit and
dfs.namenode.replication.work.multiplier.per.iteration?
{code:java}
// BlockManager#computeDatanodeWork
final int blocksToProcess = numlive
* this.blocksReplWorkMultiplier;
final int nodesToProcess = (int) Math.ceil(numlive
* this.blocksInvalidateWorkPct);
int workFound = this.computeBlockReconstructionWork(blocksToProcess); {code}
> EC: Improve performance of decommissioning dn with many ec blocks
> -
>
> Key: HDFS-16613
> URL: https://issues.apache.org/jira/browse/HDFS-16613
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec, erasure-coding, namenode
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-06-07-11-46-42-389.png,
> image-2022-06-07-17-42-16-075.png, image-2022-06-07-17-45-45-316.png,
> image-2022-06-07-17-51-04-876.png, image-2022-06-07-17-55-40-203.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In a hdfs cluster with a lot of EC blocks, decommission a dn is very slow.
> The reason is unlike replication blocks can be replicated from any dn which
> has the same block replication, the ec block have to be replicated from the
> decommissioning dn.
> The configurations dfs.namenode.replication.max-streams and
> dfs.namenode.replication.max-streams-hard-limit will limit the replication
> speed, but increase these configurations will create risk to the whole
> cluster's network. So it should add a new configuration to limit the
> decommissioning dn, distinguished from the cluster wide max-streams limit.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16613) EC: Improve performance of decommissioning dn with many ec blocks
[ https://issues.apache.org/jira/browse/HDFS-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550845#comment-17550845 ] Hiroyuki Adachi commented on HDFS-16613: Thank you for your share. How many EC blocks does the decommissioning datanode have and how many datanodes in your cluster? I'm also interested in the load (network traffic, disk I/O, etc.) of the other datanodes while decommissioning. As I mentioned above, I think the other datanodes' loads were higher due to the reconstruction tasks. Was there any impact? > EC: Improve performance of decommissioning dn with many ec blocks > - > > Key: HDFS-16613 > URL: https://issues.apache.org/jira/browse/HDFS-16613 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ec, erasure-coding, namenode >Affects Versions: 3.4.0 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Labels: pull-request-available > Attachments: image-2022-06-07-11-46-42-389.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > In a hdfs cluster with a lot of EC blocks, decommission a dn is very slow. > The reason is unlike replication blocks can be replicated from any dn which > has the same block replication, the ec block have to be replicated from the > decommissioning dn. > The configurations dfs.namenode.replication.max-streams and > dfs.namenode.replication.max-streams-hard-limit will limit the replication > speed, but increase these configurations will create risk to the whole > cluster's network. So it should add a new configuration to limit the > decommissioning dn, distinguished from the cluster wide max-streams limit. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16613) EC: Improve performance of decommissioning dn with many ec blocks
[ https://issues.apache.org/jira/browse/HDFS-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550779#comment-17550779 ] Hiroyuki Adachi commented on HDFS-16613: Hi [~caozhiqiang] , Using dfs.namenode.replication.max-streams-hard-limit is simple, but in my understanding, it makes the decommissioning node busy, and most of the EC blocks will not be replicated but reconstructed (see HDFS-14768). Since reconstruction is expensive, HDFS-8786 makes using replication for EC blocks on decommissioning node. Some people may prefer this. What do you think of this? > EC: Improve performance of decommissioning dn with many ec blocks > - > > Key: HDFS-16613 > URL: https://issues.apache.org/jira/browse/HDFS-16613 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ec, erasure-coding, namenode >Affects Versions: 3.4.0 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In a hdfs cluster with a lot of EC blocks, decommission a dn is very slow. > The reason is unlike replication blocks can be replicated from any dn which > has the same block replication, the ec block have to be replicated from the > decommissioning dn. > The configurations dfs.namenode.replication.max-streams and > dfs.namenode.replication.max-streams-hard-limit will limit the replication > speed, but increase these configurations will create risk to the whole > cluster's network. So it should add a new configuration to limit the > decommissioning dn, distinguished from the cluster wide max-streams limit. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
