[
https://issues.apache.org/jira/browse/HDDS-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509931#comment-17509931
]
Stephen O'Donnell commented on HDDS-5327:
-----------------------------------------
I've no idea if other systems use any of these ideas. HDFS does not. The entire
container concept makes this much more difficult than HDFS, which establishes a
new pipeline for each block request. Certainly on HDFS, it can take some
aspects of load into account, but if you really try to overwhelm the cluster,
it will keep allocating and trying to create blocks until the cluster becomes
unusable.
I think your understanding is correct on the above points. I have not given
this area much thought for a long time now. In general, I'd be keen to try
something simple and see how it works in practice, rather than trying to do
something too complex.
> EC: WritableEcContainerProvider should dynamically adjust the open container
> groups
> -----------------------------------------------------------------------------------
>
> Key: HDDS-5327
> URL: https://issues.apache.org/jira/browse/HDDS-5327
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Stephen O'Donnell
> Priority: Major
>
> After some discussion we concluded that for any given EC policy, a minimum
> number of pipelines should be allocated so writes can be directed to
> different containers.
> The absolute maximum number of pipelines can be calculated as some function
> of the cluster nodes and disks, but we are still unsure about how to
> calculate that limit.
> The number of pipelines should be able to grow from the minimum toward the
> maximum, depending on the write load on the cluster, or perhaps the age of
> the oldest open container (to prevent too many containers remaining open for
> too long).
> The goal is to allow a sufficient number of open containers so the writes are
> spread across different disks, without having to maintain too many open
> containers on the cluster.
> If the write load is very high, there should be more open containers than if
> the write load is very low.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
