[
https://issues.apache.org/jira/browse/HDDS-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509908#comment-17509908
]
Mark Gui commented on HDDS-5327:
--------------------------------
Hi [~sodonnell] , so let me try to understand it a bit:
# For the min value, 5 is temp choice, a more proper choice should be
considered, maybe based on some calculation or fixed. But we don't want to let
the already in use DNs be overloaded, right?
# For the max value, it seems that still not decided how to calculate it. But
we don't want to let some DNs in the cluster be overloaded first, right?
# For the grow/shrink, there are 2 things to consider: static pipeline count
and dynamic open container request rate(part of the "load" here). So it sounds
that this is more than a problem of min and max, it also involves to decide a
low/high water mark as hints to grow or shrink.
Please correct me if there's misunderstandings.
BTW, I wonder if there's other existing distributed storage systems using a
similar way that you introduced above?
And does it really have benefits to performance, cluster stability or other
factors? Could you give some examples or further explanation? Thanks~
> EC: WritableEcContainerProvider should dynamically adjust the open container
> groups
> -----------------------------------------------------------------------------------
>
> Key: HDDS-5327
> URL: https://issues.apache.org/jira/browse/HDDS-5327
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Stephen O'Donnell
> Priority: Major
>
> After some discussion we concluded that for any given EC policy, a minimum
> number of pipelines should be allocated so writes can be directed to
> different containers.
> The absolute maximum number of pipelines can be calculated as some function
> of the cluster nodes and disks, but we are still unsure about how to
> calculate that limit.
> The number of pipelines should be able to grow from the minimum toward the
> maximum, depending on the write load on the cluster, or perhaps the age of
> the oldest open container (to prevent too many containers remaining open for
> too long).
> The goal is to allow a sufficient number of open containers so the writes are
> spread across different disks, without having to maintain too many open
> containers on the cluster.
> If the write load is very high, there should be more open containers than if
> the write load is very low.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
