[
https://issues.apache.org/jira/browse/HDDS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Gui updated HDDS-5413:
---------------------------
Description:
For now, ReplicationManager process all containers at once, this will
potentially bring healthy load to datanodes if there are a lot of containers to
be replicated/deleted/closed.
So it is nice to have a bound for each round, HDFS has similar settings, and
this issue tries to implement sth like
'dfs.block.misreplication.processing.limit: 10000' in HDFS.
This is just a limit on the number of containers to be processed, note that
ReplicationManager count each container as processed no mater it is under
replicated or over replicated or good
was:
For now, ReplicationManager process all containers at once, this will
potentially bring healthy load to datanodes if there are a lot of containers to
be replicated/deleted/closed.
So it is nice to have a bound for each round, HDFS has similar settings, and
this issue tries to implement sth like
'dfs.block.misreplication.processing.limit: 10000' in HDFS.
HDFS has 128MB block by default while Ozone has 5GB container by default, so
128MB * 10000 / (5 * 1024) = 250 should be reasonable.
> Limit num of containers to process per round for ReplicationManager.
> --------------------------------------------------------------------
>
> Key: HDDS-5413
> URL: https://issues.apache.org/jira/browse/HDDS-5413
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Mark Gui
> Assignee: Mark Gui
> Priority: Major
>
> For now, ReplicationManager process all containers at once, this will
> potentially bring healthy load to datanodes if there are a lot of containers
> to be replicated/deleted/closed.
> So it is nice to have a bound for each round, HDFS has similar settings, and
> this issue tries to implement sth like
> 'dfs.block.misreplication.processing.limit: 10000' in HDFS.
>
> This is just a limit on the number of containers to be processed, note that
> ReplicationManager count each container as processed no mater it is under
> replicated or over replicated or good
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
