[
https://issues.apache.org/jira/browse/HDDS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17611:
-------------------------------------
Assignee: Andrey Yarovoy
> size the per-datanode RATIS/THREE pipeline limit from the number of data
> volumes
> --------------------------------------------------------------------------------
>
> Key: HDDS-17611
> URL: https://issues.apache.org/jira/browse/HDDS-17611
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> ## Description
> **Today** (`SCMNodeManager.pipelineLimit`):
> - If `ozone.scm.datanode.pipeline.limit` > 0 (default **2**), that's the
> limit for every datanode.
> - Only if it is set to 0, the limit is `ozone.scm.pipeline.per.metadata.disk`
> (default 2) × the number of
> **metadata** volumes the datanode reports, that is, its Raft log directories.
> Most datanodes have exactly one.
> So the default is 2 pipelines per datanode whatever its size, and the
> alternative gives 2 as well on the usual layout
> of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one
> with a single disk, unless an operator
> sets the limit by hand. Our cluster, for example, had
> `ozone.scm.datanode.pipeline.limit = 20` set by its
> deployment tooling, not derived from anything.
> The open-container count per pipeline already scales with the data volumes
> (HDDS-3700:
> `1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) /
> pipelineLimit`). The pipeline count is the
> one piece that doesn't.
> **Proposal:**
> - **A new setting,** `ozone.scm.pipeline.per.data.volume` (a decimal, default
> 0 = off). When it is > 0 and
> `ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1,
> round(per.data.volume × healthy data volumes))`.
> - **Keep the metadata-volume rule** as it is when `per.data.volume` is 0, for
> compatibility.
> - **Document how to pick the value** (HDDS-15246): the Raft log disk's
> capacity for many concurrent groups, datanode
> memory per group, and the recovery time after a datanode failure all grow
> with the number of pipelines.
> - **Possibly, as a follow-up,** make the volume-based limit the default
> (HDDS-9516).
> Because healthy volumes are re-reported, a datanode that loses disks would
> get a lower limit. Existing pipelines above
> the new limit stay until they close; no forced teardown.
> **Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO 3-way
> writes):**
> | per-datanode limit | pipelines (per datanode) | fitted 3-way write rate |
> largest write (1000 × 1 GB) |
> |---|---|---|---|
> | 20 (1.25 per data volume) | 71 (5–20) | 6.9 GB/s | 5.9 GB/s |
> | 40 (2.5 per data volume) | 154 (28–40) | 9.6 GB/s | 7.8 GB/s |
> Part of that gain is the layout: at 20, two datanodes were stranded at 5 and
> 8 pipelines, see HDDS-6225. After
> separating it out, each datanode still moved about 12% more data at 40. The
> data HDDs went from about 66% to 80% busy
> on the largest write, so more pipelines per datanode put the disks to better
> use. We didn't test a value above 40.
> **Tests:**
> - `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes;
> volume failure lowers the limit.
> - The fallback order: `datanode.pipeline.limit` > 0 wins, then
> `per.data.volume`, then `per.metadata.disk`.
> - `openContainerLimit` stays consistent with the new limit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
