Yashaswini G A created HDDS-15799:
-------------------------------------
Summary: OPEN RATIS pipelines retain stale DatanodeDetails without
RATIS_DATASTREAM after DataStream is enabled
Key: HDDS-15799
URL: https://issues.apache.org/jira/browse/HDDS-15799
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Reporter: Yashaswini G A
Enabling Ratis DataStream on an existing cluster can leave OPEN pipelines whose
member DatanodeDetails do not include {{{}RATIS_DATASTREAM{}}}, even after
datanodes are listening on the DataStream port and SCM’s node registry shows
the port correctly. O3FS/Ozone client DataStream writes then fail at runtime.
When {{ozone.fs.datastream.enabled=true}} and a write exceeds the
auto-threshold, the client uses BlockDataStreamOutput, which requires every
pipeline member to have an explicit {{RATIS_DATASTREAM}} port (hasPort() — no
fallback to the RATIS port).
Pipeline members are a frozen snapshot of DatanodeDetails captured at pipeline
creation. SCM does not refresh member ports when:
* {{dfs.container.ratis.datastream.enabled}} is enabled after pipelines were
created, or
* datanodes later gain the {{RATIS_DATASTREAM}} port.
SCM can still allocate blocks/containers on these OPEN pipelines. The client
fails with:
{{java.io.IOException: RATIS_DATASTREAM port is missing for datanode
f475cb4a-f60a-4437-b102-d317b8e6118c(ccycloud-4.quasar-fqjoax.root.comops.site/10.140.80.132)
in pipeline Pipeline-05ad9496-a228-4644-ab9f-ea6dc0426d73; datastream is
disabled for this pipeline"}}
h3. Steps to reproduce
# Start cluster with default configs
{{dfs.container.ratis.datastream.enabled=false}} and
{{ozone.fs.datastream.enabled=false}}
# SCM may have created OPEN RATIS pipelines (RF=3).
# Enable datastream configs in step 1 and restart ozone service.
# Confirm DNs listen on DataStream port and SCM datanode list shows
{{{}RATIS_DATASTREAM{}}}.
# Perform O3FS write > {{ozone.fs.datastream.auto.threshold}} (e.g. 20 MB
{{ozone fs -put}} / {{{}O3fs.put{}}}).
h3. Actual behavior
Write fails if block allocation uses a pre-existing OPEN pipeline whose members
lack {{{}RATIS_DATASTREAM{}}}.
h3. Expected behavior
Either:
* OPEN pipelines are refreshed/recreated so members include current ports, or
* SCM/OM avoids allocating DataStream writes to incompatible pipelines, or
* A clear upgrade path automatically closes stale OPEN RATIS pipelines when
DataStream is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
