[
https://issues.apache.org/jira/browse/HDFS-16846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638808#comment-17638808
]
ASF GitHub Bot commented on HDFS-16846:
---------------------------------------
lfxy commented on code in PR #5143:
URL: https://github.com/apache/hadoop/pull/5143#discussion_r1032739690
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##########
@@ -1825,28 +1825,48 @@ public DatanodeCommand[]
handleHeartbeat(DatanodeRegistration nodeReg,
// Allocate _approximately_ maxTransfers pending tasks to DataNode.
// NN chooses pending tasks based on the ratio between the lengths of
// replication and erasure-coded block queues.
- int totalReplicateBlocks = nodeinfo.getNumberOfReplicateBlocks();
- int totalECBlocks = nodeinfo.getNumberOfBlocksToBeErasureCoded();
- int totalBlocks = totalReplicateBlocks + totalECBlocks;
+ int replicationBlocks = nodeinfo.getNumberOfReplicateBlocks();
+ int ecReplicatedBlocks = nodeinfo.getNumberOfECReplicatedBlocks();
+ int ecReconstructedBlocks = nodeinfo.getNumberOfBlocksToBeErasureCoded();
+ int totalBlocks = replicationBlocks + ecReplicatedBlocks +
ecReconstructedBlocks;
if (totalBlocks > 0) {
- int maxTransfers;
+ int maxReplicationTransfers = blockManager.getMaxReplicationStreams()
+ - xmitsInProgress;
+ int maxECReplicatedTransfers;
+ int maxECReconstructedTransfers;
Review Comment:
Yes, it's right.
> EC: Only EC blocks should be effected by max-streams-hard-limit configuration
> -----------------------------------------------------------------------------
>
> Key: HDFS-16846
> URL: https://issues.apache.org/jira/browse/HDFS-16846
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec
> Affects Versions: 3.4.0
> Reporter: caozhiqiang
> Assignee: caozhiqiang
> Priority: Major
> Labels: pull-request-available
>
> In [HDFS-16613|https://issues.apache.org/jira/browse/HDFS-16613], the
> dfs.namenode.replication.max-streams-hard-limit configuration will only
> affect decommissioning DataNode, but will not distinguish between replication
> blocks and EC blocks. Even if DataNodes have only replication files, they
> will always generate high network traffic. So this configuration should only
> effect EC blocks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
