[
https://issues.apache.org/jira/browse/HDDS-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swaminathan Balachandran updated HDDS-11099:
--------------------------------------------
Description:
If *ozone.replication.type* is EC, background pipeline creator will not create
any pipeline for ratis. When a block allocate request comes to the SCM, the SCM
will try to find an existing container to allocate the block. If it is not able
to find a suitable container this will lead to creating a ratis container on
the fly. While trying to create a container it will look for existing pipelines
in the system and if there is no pipeline, a pipeline will be created and if
there is only a single pipeline in the system, the same pipeline will be used
to create a container over and over again for all blocks. This will happen till
the DN doesn't run out of space and another pipeline would be created. But this
would lead to writes getting choked to just 3 nodes at a time.
[https://github.com/apache/ozone/blob/c1e9e9180960777ac99e6483515e8b76e585f5d0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableRatisContainerProvider.java#L163]
cc [~umamaheswararao]
was:
If *ozone.replication.type* is EC, background pipeline creator will not create
any pipeline for ratis. When a block allocate request comes to the SCM, the SCM
will try to find an existing container to allocate the block. If it is not able
to find a suitable container this will lead to creating a ratis container on
the fly. While trying to create a container it will look for existing pipelines
in the system and if there is no pipeline, a pipeline will be created and if
there is only a single pipeline in the system, the same pipeline will be used
to create a container over and over again for all blocks. This will happen till
the DN doesn't run out of space and another pipeline would be created. But this
would lead to writes getting choked to just 3 nodes at a time.
https://github.com/apache/ozone/blob/c1e9e9180960777ac99e6483515e8b76e585f5d0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableRatisContainerProvider.java#L163
> Ozone will have only one pipeline for ratis keys
> ------------------------------------------------
>
> Key: HDDS-11099
> URL: https://issues.apache.org/jira/browse/HDDS-11099
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Swaminathan Balachandran
> Assignee: Swaminathan Balachandran
> Priority: Major
>
> If *ozone.replication.type* is EC, background pipeline creator will not
> create any pipeline for ratis. When a block allocate request comes to the
> SCM, the SCM will try to find an existing container to allocate the block. If
> it is not able to find a suitable container this will lead to creating a
> ratis container on the fly. While trying to create a container it will look
> for existing pipelines in the system and if there is no pipeline, a pipeline
> will be created and if there is only a single pipeline in the system, the
> same pipeline will be used to create a container over and over again for all
> blocks. This will happen till the DN doesn't run out of space and another
> pipeline would be created. But this would lead to writes getting choked to
> just 3 nodes at a time.
> [https://github.com/apache/ozone/blob/c1e9e9180960777ac99e6483515e8b76e585f5d0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableRatisContainerProvider.java#L163]
> cc [~umamaheswararao]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
