[
https://issues.apache.org/jira/browse/HDDS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17611:
----------------------------------
Description:
*{*}Today{*}* (`SCMNodeManager.pipelineLimit`):
- If `ozone.scm.datanode.pipeline.limit` > 0 (default *{*}2{*}*), that's the
limit for every datanode.
- Only if it is set to 0, the limit is `ozone.scm.pipeline.per.metadata.disk`
(default 2) × the number of
*{*}metadata{*}* volumes the datanode reports, that is, its Raft log
directories. Most datanodes have exactly one.
So the default is 2 pipelines per datanode whatever its size, and the
alternative gives 2 as well on the usual layout
of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one with
a single disk, unless an operator
sets the limit by hand. Our cluster, for example, had
`ozone.scm.datanode.pipeline.limit = 20` set by its deployment tooling, not
derived from anything.
The open-container count per pipeline already scales with the data volumes
(HDDS-3700:
`1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) /
pipelineLimit`). The pipeline count is the one piece that doesn't.
*{*}Proposal:{*}*
- *{*}A new setting,{*}* `ozone.scm.pipeline.per.data.volume` (a decimal,
default 0 = off). When it is > 0 and
`ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1,
round(per.data.volume × healthy data volumes))`.
- *{*}Keep the metadata-volume rule{*}* as it is when `per.data.volume` is 0,
for compatibility.
- *{*}Document how to pick the value{*}* (HDDS-15246): the Raft log disk's
capacity for many concurrent groups, datanode memory per group, and the
recovery time after a datanode failure all grow with the number of pipelines.
- *{*}Possibly, as a follow-up,{*}* make the volume-based limit the default
(HDDS-9516).
Because healthy volumes are re-reported, a datanode that loses disks would get
a lower limit. Existing pipelines above
the new limit stay until they close; no forced teardown.
*{*}Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO 3-way
writes):{*}*
|per-datanode limit|pipelines (per datanode)|fitted 3-way write rate|largest
write (1000 × 1 GB)|
|—|—|—|—|
|20 (1.25 per data volume)|71 (5–20)|6.9 GB/s|5.9 GB/s|
|40 (2.5 per data volume)|154 (28–40)|9.6 GB/s|7.8 GB/s|
Part of that gain is the layout: at 20, two datanodes were stranded at 5 and 8
pipelines, see HDDS-6225. After separating it out, each datanode still moved
about 12% more data at 40. The data HDDs went from about 66% to 80% busy on the
largest write, so more pipelines per datanode put the disks to better use. We
didn't test a value above 40.
*{*}Tests:{*}*
- `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes;
volume failure lowers the limit.
- The fallback order: `datanode.pipeline.limit` > 0 wins, then
`per.data.volume`, then `per.metadata.disk`.
- `openContainerLimit` stays consistent with the new limit.
was:
## Description
**Today** (`SCMNodeManager.pipelineLimit`):
- If `ozone.scm.datanode.pipeline.limit` > 0 (default **2**), that's the limit
for every datanode.
- Only if it is set to 0, the limit is `ozone.scm.pipeline.per.metadata.disk`
(default 2) × the number of
**metadata** volumes the datanode reports, that is, its Raft log directories.
Most datanodes have exactly one.
So the default is 2 pipelines per datanode whatever its size, and the
alternative gives 2 as well on the usual layout
of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one with
a single disk, unless an operator
sets the limit by hand. Our cluster, for example, had
`ozone.scm.datanode.pipeline.limit = 20` set by its
deployment tooling, not derived from anything.
The open-container count per pipeline already scales with the data volumes
(HDDS-3700:
`1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) /
pipelineLimit`). The pipeline count is the
one piece that doesn't.
**Proposal:**
- **A new setting,** `ozone.scm.pipeline.per.data.volume` (a decimal, default 0
= off). When it is > 0 and
`ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1,
round(per.data.volume × healthy data volumes))`.
- **Keep the metadata-volume rule** as it is when `per.data.volume` is 0, for
compatibility.
- **Document how to pick the value** (HDDS-15246): the Raft log disk's capacity
for many concurrent groups, datanode
memory per group, and the recovery time after a datanode failure all grow with
the number of pipelines.
- **Possibly, as a follow-up,** make the volume-based limit the default
(HDDS-9516).
Because healthy volumes are re-reported, a datanode that loses disks would get
a lower limit. Existing pipelines above
the new limit stay until they close; no forced teardown.
**Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO 3-way
writes):**
| per-datanode limit | pipelines (per datanode) | fitted 3-way write rate |
largest write (1000 × 1 GB) |
|---|---|---|---|
| 20 (1.25 per data volume) | 71 (5–20) | 6.9 GB/s | 5.9 GB/s |
| 40 (2.5 per data volume) | 154 (28–40) | 9.6 GB/s | 7.8 GB/s |
Part of that gain is the layout: at 20, two datanodes were stranded at 5 and 8
pipelines, see HDDS-6225. After
separating it out, each datanode still moved about 12% more data at 40. The
data HDDs went from about 66% to 80% busy
on the largest write, so more pipelines per datanode put the disks to better
use. We didn't test a value above 40.
**Tests:**
- `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes;
volume failure lowers the limit.
- The fallback order: `datanode.pipeline.limit` > 0 wins, then
`per.data.volume`, then `per.metadata.disk`.
- `openContainerLimit` stays consistent with the new limit.
> size the per-datanode RATIS/THREE pipeline limit from the number of data
> volumes
> --------------------------------------------------------------------------------
>
> Key: HDDS-17611
> URL: https://issues.apache.org/jira/browse/HDDS-17611
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> *{*}Today{*}* (`SCMNodeManager.pipelineLimit`):
> - If `ozone.scm.datanode.pipeline.limit` > 0 (default *{*}2{*}*), that's the
> limit for every datanode.
> - Only if it is set to 0, the limit is
> `ozone.scm.pipeline.per.metadata.disk` (default 2) × the number of
> *{*}metadata{*}* volumes the datanode reports, that is, its Raft log
> directories. Most datanodes have exactly one.
> So the default is 2 pipelines per datanode whatever its size, and the
> alternative gives 2 as well on the usual layout
> of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one
> with a single disk, unless an operator
> sets the limit by hand. Our cluster, for example, had
> `ozone.scm.datanode.pipeline.limit = 20` set by its deployment tooling, not
> derived from anything.
> The open-container count per pipeline already scales with the data volumes
> (HDDS-3700:
> `1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) /
> pipelineLimit`). The pipeline count is the one piece that doesn't.
> *{*}Proposal:{*}*
> - *{*}A new setting,{*}* `ozone.scm.pipeline.per.data.volume` (a decimal,
> default 0 = off). When it is > 0 and
> `ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1,
> round(per.data.volume × healthy data volumes))`.
> - *{*}Keep the metadata-volume rule{*}* as it is when `per.data.volume` is
> 0, for compatibility.
> - *{*}Document how to pick the value{*}* (HDDS-15246): the Raft log disk's
> capacity for many concurrent groups, datanode memory per group, and the
> recovery time after a datanode failure all grow with the number of pipelines.
> - *{*}Possibly, as a follow-up,{*}* make the volume-based limit the default
> (HDDS-9516).
> Because healthy volumes are re-reported, a datanode that loses disks would
> get a lower limit. Existing pipelines above
> the new limit stay until they close; no forced teardown.
> *{*}Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO
> 3-way writes):{*}*
> |per-datanode limit|pipelines (per datanode)|fitted 3-way write rate|largest
> write (1000 × 1 GB)|
> |—|—|—|—|
> |20 (1.25 per data volume)|71 (5–20)|6.9 GB/s|5.9 GB/s|
> |40 (2.5 per data volume)|154 (28–40)|9.6 GB/s|7.8 GB/s|
> Part of that gain is the layout: at 20, two datanodes were stranded at 5 and
> 8 pipelines, see HDDS-6225. After separating it out, each datanode still
> moved about 12% more data at 40. The data HDDs went from about 66% to 80%
> busy on the largest write, so more pipelines per datanode put the disks to
> better use. We didn't test a value above 40.
> *{*}Tests:{*}*
> - `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes;
> volume failure lowers the limit.
> - The fallback order: `datanode.pipeline.limit` > 0 wins, then
> `per.data.volume`, then `per.metadata.disk`.
> - `openContainerLimit` stays consistent with the new limit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]