[
https://issues.apache.org/jira/browse/HDFS-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093119#comment-18093119
]
ASF GitHub Bot commented on HDFS-17937:
---------------------------------------
Hexiaoqiao commented on PR #8559:
URL: https://github.com/apache/hadoop/pull/8559#issuecomment-4862218353
Thanks @huangzhaobo99 for your works. Make sense to me. One concern, If the
fsimage files may amass to large numbers because it can't upload to other
namenodes on time when enable this configuration to end user by default?
> Limit fsimage transfer concurrency during checkpoint with multiple NameNodes.
> -----------------------------------------------------------------------------
>
> Key: HDFS-17937
> URL: https://issues.apache.org/jira/browse/HDFS-17937
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Zhaobo Huang
> Assignee: Zhaobo Huang
> Priority: Major
> Labels: pull-request-available
>
> This patch adds dfs.namenode.checkpoint.parallel.upload.max-threads to cap
> fsimage upload concurrency in StandbyCheckpointer when parallel upload is
> enabled.
> The limit is needed to avoid excessive simultaneous fsimage uploads in
> multi-NameNode (9 Observer NN) HA deployments, which can otherwise increase
> CPU/Network/Disk pressure and impact checkpoint/upload stability.
> Effective concurrency is now min(remoteNNAddresses.size(), max-threads), with
> config/docs updates in DFSConfigKeys, CheckpointConf, and hdfs-default.xml,
> plus validation that max-threads > 0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
