[
https://issues.apache.org/jira/browse/HDFS-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093172#comment-18093172
]
ASF GitHub Bot commented on HDFS-17937:
---------------------------------------
Hexiaoqiao commented on PR #8559:
URL: https://github.com/apache/hadoop/pull/8559#issuecomment-4863461336
My bad, I want to say that the checkpoint could not execute as expect
because checkpoint time cost will enlarge.
Consider case: 1 ANN + 1 SBN + 6 Observer, with configuration checkpoint
every hour and `maxThreads=1` when checkpoint and fsimage file size is 100GB.
The whole time cost could be like the following,
100GB*(1024MB/GB)/(100MB/s) * 8 =136min where 8 = 1 (dump) + 7(upload)
Here 136min>60min, and checkpoint will postpone about 2 hours later which is
not expect, right?
Thanks.
> Limit fsimage transfer concurrency during checkpoint with multiple NameNodes.
> -----------------------------------------------------------------------------
>
> Key: HDFS-17937
> URL: https://issues.apache.org/jira/browse/HDFS-17937
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Zhaobo Huang
> Assignee: Zhaobo Huang
> Priority: Major
> Labels: pull-request-available
>
> This patch adds dfs.namenode.checkpoint.parallel.upload.max-threads to cap
> fsimage upload concurrency in StandbyCheckpointer when parallel upload is
> enabled.
> The limit is needed to avoid excessive simultaneous fsimage uploads in
> multi-NameNode (9 Observer NN) HA deployments, which can otherwise increase
> CPU/Network/Disk pressure and impact checkpoint/upload stability.
> Effective concurrency is now min(remoteNNAddresses.size(), max-threads), with
> config/docs updates in DFSConfigKeys, CheckpointConf, and hdfs-default.xml,
> plus validation that max-threads > 0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
