[
https://issues.apache.org/jira/browse/HDDS-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-15799:
-------------------------------------
Assignee: Andrey Yarovoy
> OPEN RATIS pipelines retain stale DatanodeDetails without RATIS_DATASTREAM
> after DataStream is enabled
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15799
> URL: https://issues.apache.org/jira/browse/HDDS-15799
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Reporter: Yashaswini G A
> Assignee: Andrey Yarovoy
> Priority: Major
>
> Enabling Ratis DataStream on an existing cluster can leave OPEN pipelines
> whose member DatanodeDetails do not include {{{}RATIS_DATASTREAM{}}}, even
> after datanodes are listening on the DataStream port and SCM’s node registry
> shows the port correctly. O3FS/Ozone client DataStream writes then fail at
> runtime.
> When {{ozone.fs.datastream.enabled=true}} and a write exceeds the
> auto-threshold, the client uses BlockDataStreamOutput, which requires every
> pipeline member to have an explicit {{RATIS_DATASTREAM}} port (hasPort() — no
> fallback to the RATIS port).
> Pipeline members are a frozen snapshot of DatanodeDetails captured at
> pipeline creation. SCM does not refresh member ports when:
> * {{dfs.container.ratis.datastream.enabled}} is enabled after pipelines were
> created, or
> * datanodes later gain the {{RATIS_DATASTREAM}} port.
> SCM can still allocate blocks/containers on these OPEN pipelines. The client
> fails with:
>
> {{java.io.IOException: RATIS_DATASTREAM port is missing for datanode
> f475cb4a-f60a-4437-b102-d317b8e6118c(ccycloud-4.quasar-fqjoax.root.comops.site/10.140.80.132)
> in pipeline Pipeline-05ad9496-a228-4644-ab9f-ea6dc0426d73; datastream is
> disabled for this pipeline"}}
>
>
> h3. Steps to reproduce
> # Start cluster with default configs
> {{dfs.container.ratis.datastream.enabled=false}} and
> {{ozone.fs.datastream.enabled=false}}
> # SCM may have created OPEN RATIS pipelines (RF=3).
> # Enable datastream configs in step 1 and restart ozone service.
> # Confirm DNs listen on DataStream port and SCM datanode list shows
> {{{}RATIS_DATASTREAM{}}}.
> # Perform O3FS write > {{ozone.fs.datastream.auto.threshold}} (e.g. 20 MB
> {{ozone fs -put}} / {{{}O3fs.put{}}}).
> h3. Actual behavior
> Write fails if block allocation uses a pre-existing OPEN pipeline whose
> members lack {{{}RATIS_DATASTREAM{}}}.
> h3. Expected behavior
> Either:
> * OPEN pipelines are refreshed/recreated so members include current ports, or
> * SCM/OM avoids allocating DataStream writes to incompatible pipelines, or
> * A clear upgrade path automatically closes stale OPEN RATIS pipelines when
> DataStream is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
