[
https://issues.apache.org/jira/browse/HDFS-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhaobo Huang updated HDFS-17937:
--------------------------------
Description:
This patch adds dfs.namenode.checkpoint.parallel.upload.max-threads to cap
fsimage upload concurrency in StandbyCheckpointer when parallel upload is
enabled.
The limit is needed to avoid excessive simultaneous fsimage uploads in
multi-NameNode (9 Observer NN) HA deployments, which can otherwise increase
CPU/Network/Disk pressure and impact checkpoint/upload stability.
Effective concurrency is now min(remoteNNAddresses.size(), max-threads), with
config/docs updates in DFSConfigKeys, CheckpointConf, and hdfs-default.xml,
plus validation that max-threads > 0.
> Limit fsimage transfer concurrency during checkpoint with multiple NameNodes.
> -----------------------------------------------------------------------------
>
> Key: HDFS-17937
> URL: https://issues.apache.org/jira/browse/HDFS-17937
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Zhaobo Huang
> Assignee: Zhaobo Huang
> Priority: Major
> Labels: pull-request-available
>
> This patch adds dfs.namenode.checkpoint.parallel.upload.max-threads to cap
> fsimage upload concurrency in StandbyCheckpointer when parallel upload is
> enabled.
> The limit is needed to avoid excessive simultaneous fsimage uploads in
> multi-NameNode (9 Observer NN) HA deployments, which can otherwise increase
> CPU/Network/Disk pressure and impact checkpoint/upload stability.
> Effective concurrency is now min(remoteNNAddresses.size(), max-threads), with
> config/docs updates in DFSConfigKeys, CheckpointConf, and hdfs-default.xml,
> plus validation that max-threads > 0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
