[
https://issues.apache.org/jira/browse/HDFS-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093159#comment-18093159
]
ASF GitHub Bot commented on HDFS-17937:
---------------------------------------
huangzhaobo99 commented on PR #8559:
URL: https://github.com/apache/hadoop/pull/8559#issuecomment-4863142748
> Thanks @huangzhaobo99 for your works. Make sense to me. One concern, If
the fsimage files may amass to large numbers because it can't upload to other
namenodes on time when enable this configuration to end user by default?
@Hexiaoqiao Thanks for review.
I updated the concurrency policy in code as follows:
1. When parallel upload is enabled and maxThreads == 0 (default), poolSize
is set to remoteNNAddresses.size(), preserving the original behavior.
2. When maxThreads > 0, poolSize is set to
Math.min(remoteNNAddresses.size(), maxThreads), so concurrency is capped by the
configured maximum.
Therefore, when there are many Observer NN nodes, concurrency can be limited
based on the actual deployment scenario, It will not easily lead to fsimage
amass.
> Limit fsimage transfer concurrency during checkpoint with multiple NameNodes.
> -----------------------------------------------------------------------------
>
> Key: HDFS-17937
> URL: https://issues.apache.org/jira/browse/HDFS-17937
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Zhaobo Huang
> Assignee: Zhaobo Huang
> Priority: Major
> Labels: pull-request-available
>
> This patch adds dfs.namenode.checkpoint.parallel.upload.max-threads to cap
> fsimage upload concurrency in StandbyCheckpointer when parallel upload is
> enabled.
> The limit is needed to avoid excessive simultaneous fsimage uploads in
> multi-NameNode (9 Observer NN) HA deployments, which can otherwise increase
> CPU/Network/Disk pressure and impact checkpoint/upload stability.
> Effective concurrency is now min(remoteNNAddresses.size(), max-threads), with
> config/docs updates in DFSConfigKeys, CheckpointConf, and hdfs-default.xml,
> plus validation that max-threads > 0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
