[
https://issues.apache.org/jira/browse/HDDS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Gui updated HDDS-5413:
---------------------------
Description:
For now, ReplicationManager process all containers at once, this will
potentially bring healthy load to datanodes if there are a lot of containers to
be replicated/deleted/closed.
So it is nice to have a bound for each round, HDFS has similar settings, and
this issue tries to implement sth like
'dfs.block.misreplication.processing.limit: 10000' in HDFS.
This is just a limit on the number of containers to be processed, note that
ReplicationManager count each container as processed no mater it is under
replicated or over replicated or good. And all the cmds are queued and will be
sent with heartbeats.
So this limit does not directly limit the data to be replicated, just to have a
basic throttling for ReplicationManager in-memory processing, so 10000 should
be good for ozone as well.
And we should add more throttling limits in the future.
was:
For now, ReplicationManager process all containers at once, this will
potentially bring healthy load to datanodes if there are a lot of containers to
be replicated/deleted/closed.
So it is nice to have a bound for each round, HDFS has similar settings, and
this issue tries to implement sth like
'dfs.block.misreplication.processing.limit: 10000' in HDFS.
This is just a limit on the number of containers to be processed, note that
ReplicationManager count each container as processed no mater it is under
replicated or over replicated or good. And all the cmds are queued and will be
sent with heartbeats.
So this limit does not directly limit the data to be replicated, so 10000
should be good for ozone as well.
And we should add more throttling limits in the future.
> Limit num of containers to process per round for ReplicationManager.
> --------------------------------------------------------------------
>
> Key: HDDS-5413
> URL: https://issues.apache.org/jira/browse/HDDS-5413
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Mark Gui
> Assignee: Mark Gui
> Priority: Major
>
> For now, ReplicationManager process all containers at once, this will
> potentially bring healthy load to datanodes if there are a lot of containers
> to be replicated/deleted/closed.
> So it is nice to have a bound for each round, HDFS has similar settings, and
> this issue tries to implement sth like
> 'dfs.block.misreplication.processing.limit: 10000' in HDFS.
>
> This is just a limit on the number of containers to be processed, note that
> ReplicationManager count each container as processed no mater it is under
> replicated or over replicated or good. And all the cmds are queued and will
> be sent with heartbeats.
> So this limit does not directly limit the data to be replicated, just to have
> a basic throttling for ReplicationManager in-memory processing, so 10000
> should be good for ozone as well.
> And we should add more throttling limits in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
