[ 
https://issues.apache.org/jira/browse/HDDS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17611:
----------------------------------
    Description: 
*{*}Today{*}* (`SCMNodeManager.pipelineLimit`):
 - If `ozone.scm.datanode.pipeline.limit` > 0 (default *{*}2{*}*), that's the 
limit for every datanode.
 - Only if it is set to 0, the limit is `ozone.scm.pipeline.per.metadata.disk` 
(default 2) × the number of
*{*}metadata{*}* volumes the datanode reports, that is, its Raft log 
directories. Most datanodes have exactly one.

So the default is 2 pipelines per datanode whatever its size, and the 
alternative gives 2 as well on the usual layout
of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one with 
a single disk, unless an operator
sets the limit by hand. Our cluster, for example, had 
`ozone.scm.datanode.pipeline.limit = 20` set by its deployment tooling, not 
derived from anything.

The open-container count per pipeline already scales with the data volumes 
(HDDS-3700:
`1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) / 
pipelineLimit`). The pipeline count is the one piece that doesn't.

*{*}Proposal:{*}*
 - *{*}A new setting,{*}* `ozone.scm.pipeline.per.data.volume` (a decimal, 
default 0 = off). When it is > 0 and
`ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1, 
round(per.data.volume × healthy data volumes))`.
 - *{*}Keep the metadata-volume rule{*}* as it is when `per.data.volume` is 0, 
for compatibility.
 - *{*}Document how to pick the value{*}* (HDDS-15246): the Raft log disk's 
capacity for many concurrent groups, datanode memory per group, and the 
recovery time after a datanode failure all grow with the number of pipelines.
 - *{*}Possibly, as a follow-up,{*}* make the volume-based limit the default 
(HDDS-9516).

Because healthy volumes are re-reported, a datanode that loses disks would get 
a lower limit. Existing pipelines above
the new limit stay until they close; no forced teardown.

*{*}Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO 3-way 
writes):{*}*
|per-datanode limit|pipelines (per datanode)|fitted 3-way write rate|largest 
write (1000 × 1 GB)|
|—|—|—|—|
|20 (1.25 per data volume)|71 (5–20)|6.9 GB/s|5.9 GB/s|
|40 (2.5 per data volume)|154 (28–40)|9.6 GB/s|7.8 GB/s|

Part of that gain is the layout: at 20, two datanodes were stranded at 5 and 8 
pipelines, see HDDS-6225. After separating it out, each datanode still moved 
about 12% more data at 40. The data HDDs went from about 66% to 80% busy on the 
largest write, so more pipelines per datanode put the disks to better use. We 
didn't test a value above 40.

*{*}Tests:{*}*
 - `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes; 
volume failure lowers the limit.
 - The fallback order: `datanode.pipeline.limit` > 0 wins, then 
`per.data.volume`, then `per.metadata.disk`.
 - `openContainerLimit` stays consistent with the new limit.

  was:
## Description

**Today** (`SCMNodeManager.pipelineLimit`):
- If `ozone.scm.datanode.pipeline.limit` > 0 (default **2**), that's the limit 
for every datanode.
- Only if it is set to 0, the limit is `ozone.scm.pipeline.per.metadata.disk` 
(default 2) × the number of
**metadata** volumes the datanode reports, that is, its Raft log directories. 
Most datanodes have exactly one.

So the default is 2 pipelines per datanode whatever its size, and the 
alternative gives 2 as well on the usual layout
of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one with 
a single disk, unless an operator
sets the limit by hand. Our cluster, for example, had 
`ozone.scm.datanode.pipeline.limit = 20` set by its
deployment tooling, not derived from anything.

The open-container count per pipeline already scales with the data volumes 
(HDDS-3700:
`1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) / 
pipelineLimit`). The pipeline count is the
one piece that doesn't.

**Proposal:**
- **A new setting,** `ozone.scm.pipeline.per.data.volume` (a decimal, default 0 
= off). When it is > 0 and
`ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1, 
round(per.data.volume × healthy data volumes))`.
- **Keep the metadata-volume rule** as it is when `per.data.volume` is 0, for 
compatibility.
- **Document how to pick the value** (HDDS-15246): the Raft log disk's capacity 
for many concurrent groups, datanode
memory per group, and the recovery time after a datanode failure all grow with 
the number of pipelines.
- **Possibly, as a follow-up,** make the volume-based limit the default 
(HDDS-9516).

Because healthy volumes are re-reported, a datanode that loses disks would get 
a lower limit. Existing pipelines above
the new limit stay until they close; no forced teardown.

**Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO 3-way 
writes):**

| per-datanode limit | pipelines (per datanode) | fitted 3-way write rate | 
largest write (1000 × 1 GB) |
|---|---|---|---|
| 20 (1.25 per data volume) | 71 (5–20) | 6.9 GB/s | 5.9 GB/s |
| 40 (2.5 per data volume) | 154 (28–40) | 9.6 GB/s | 7.8 GB/s |

Part of that gain is the layout: at 20, two datanodes were stranded at 5 and 8 
pipelines, see HDDS-6225. After
separating it out, each datanode still moved about 12% more data at 40. The 
data HDDs went from about 66% to 80% busy
on the largest write, so more pipelines per datanode put the disks to better 
use. We didn't test a value above 40.

**Tests:**
- `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes; 
volume failure lowers the limit.
- The fallback order: `datanode.pipeline.limit` > 0 wins, then 
`per.data.volume`, then `per.metadata.disk`.
- `openContainerLimit` stays consistent with the new limit.


> size the per-datanode RATIS/THREE pipeline limit from the number of data 
> volumes
> --------------------------------------------------------------------------------
>
>                 Key: HDDS-17611
>                 URL: https://issues.apache.org/jira/browse/HDDS-17611
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: SCM
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> *{*}Today{*}* (`SCMNodeManager.pipelineLimit`):
>  - If `ozone.scm.datanode.pipeline.limit` > 0 (default *{*}2{*}*), that's the 
> limit for every datanode.
>  - Only if it is set to 0, the limit is 
> `ozone.scm.pipeline.per.metadata.disk` (default 2) × the number of
> *{*}metadata{*}* volumes the datanode reports, that is, its Raft log 
> directories. Most datanodes have exactly one.
> So the default is 2 pipelines per datanode whatever its size, and the 
> alternative gives 2 as well on the usual layout
> of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one 
> with a single disk, unless an operator
> sets the limit by hand. Our cluster, for example, had 
> `ozone.scm.datanode.pipeline.limit = 20` set by its deployment tooling, not 
> derived from anything.
> The open-container count per pipeline already scales with the data volumes 
> (HDDS-3700:
> `1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) / 
> pipelineLimit`). The pipeline count is the one piece that doesn't.
> *{*}Proposal:{*}*
>  - *{*}A new setting,{*}* `ozone.scm.pipeline.per.data.volume` (a decimal, 
> default 0 = off). When it is > 0 and
> `ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1, 
> round(per.data.volume × healthy data volumes))`.
>  - *{*}Keep the metadata-volume rule{*}* as it is when `per.data.volume` is 
> 0, for compatibility.
>  - *{*}Document how to pick the value{*}* (HDDS-15246): the Raft log disk's 
> capacity for many concurrent groups, datanode memory per group, and the 
> recovery time after a datanode failure all grow with the number of pipelines.
>  - *{*}Possibly, as a follow-up,{*}* make the volume-based limit the default 
> (HDDS-9516).
> Because healthy volumes are re-reported, a datanode that loses disks would 
> get a lower limit. Existing pipelines above
> the new limit stay until they close; no forced teardown.
> *{*}Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO 
> 3-way writes):{*}*
> |per-datanode limit|pipelines (per datanode)|fitted 3-way write rate|largest 
> write (1000 × 1 GB)|
> |—|—|—|—|
> |20 (1.25 per data volume)|71 (5–20)|6.9 GB/s|5.9 GB/s|
> |40 (2.5 per data volume)|154 (28–40)|9.6 GB/s|7.8 GB/s|
> Part of that gain is the layout: at 20, two datanodes were stranded at 5 and 
> 8 pipelines, see HDDS-6225. After separating it out, each datanode still 
> moved about 12% more data at 40. The data HDDs went from about 66% to 80% 
> busy on the largest write, so more pipelines per datanode put the disks to 
> better use. We didn't test a value above 40.
> *{*}Tests:{*}*
>  - `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes; 
> volume failure lowers the limit.
>  - The fallback order: `datanode.pipeline.limit` > 0 wins, then 
> `per.data.volume`, then `per.metadata.disk`.
>  - `openContainerLimit` stays consistent with the new limit.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to