[
https://issues.apache.org/jira/browse/HDFS-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093978#comment-18093978
]
ASF GitHub Bot commented on HDFS-17937:
---------------------------------------
Hexiaoqiao commented on PR #8559:
URL: https://github.com/apache/hadoop/pull/8559#issuecomment-4891896313
> @Hexiaoqiao Yes, this can happen. This is expected, since the concurrency
is configured to a low value, which increases checkpoint execution time. In
this scenario, we should set the parallelism to 3 rather than simply matching
the number of remote NNs. (Using 7 parallel threads is too high and
unnecessary, This is also the main problem that this patch aims to solve)
Got it. However we have changed the configuration
`dfs.namenode.checkpoint.period` logic here, I think we need tell the truth to
end users when release at least. Thanks.
> Limit fsimage transfer concurrency during checkpoint with multiple NameNodes.
> -----------------------------------------------------------------------------
>
> Key: HDFS-17937
> URL: https://issues.apache.org/jira/browse/HDFS-17937
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Zhaobo Huang
> Assignee: Zhaobo Huang
> Priority: Major
> Labels: pull-request-available
>
> This patch adds dfs.namenode.checkpoint.parallel.upload.max-threads to cap
> fsimage upload concurrency in StandbyCheckpointer when parallel upload is
> enabled.
> The limit is needed to avoid excessive simultaneous fsimage uploads in
> multi-NameNode (9 Observer NN) HA deployments, which can otherwise increase
> CPU/Network/Disk pressure and impact checkpoint/upload stability.
> Effective concurrency is now min(remoteNNAddresses.size(), max-threads), with
> config/docs updates in DFSConfigKeys, CheckpointConf, and hdfs-default.xml,
> plus validation that max-threads > 0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
