szetszwo commented on code in PR #11982:
URL: https://github.com/apache/ozone/pull/11982#discussion_r4621142956


##########
streaming-without-raft.md:
##########
@@ -0,0 +1,578 @@
+# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication)
+
+| Field   | Value                                     |
+|---------|-------------------------------------------|
+| Status  | Draft 2026-11-02                          |
+| Author  | Tsz-Wo Nicholas Sze, Rui Wang             |
+| AI      | Claude                                    |
+| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093 |
+
+---
+
+## 1. Executive Summary & Goals
+
+### 1.1 Background
+
+Apache Ozone currently has three write pipelines:
+
+| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per 
block | Replication type |
+|---|---|---|---|---|
+| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft 
| 1 per chunk + 1 per PutBlock | `RATIS` |
+| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via 
Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock | 
`RATIS` |
+| **EC** | Direct gRPC writes of data and parity chunks to each datanode | 
`PutBlock` direct to each datanode | 0 | `EC` |
+
+In V1 and V2, every write is ordered by the Raft leader and written to the 
Raft log, which costs:
+
+- **Raft log I/O.** Each entry is written to the Raft log on every datanode,
+  in addition to the container data and RocksDB metadata.
+- **Leader CPU.** The leader is a hotspot for ordering, replicating and 
applying log entries.
+- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and 
apply.
+
+Raft totally orders writes **across blocks in a container**.
+Ozone blocks have a single writer and are immutable after commit, so that 
ordering is not needed for block data.
+
+### 1.2 Proposal
+
+**Write Pipeline V3** sends **both StreamData and PutBlock over Ratis 
DataStream** and writes **no Raft log entries for any write**.
+V3 is identified by the previously unused `ReplicationType.CHAINED` (`= 3` in 
`hdds.proto`):
+
+- **Keys and containers.** CHAINED keys are written to CHAINED containers.
+- **Pipelines.** CHAINED containers live on dedicated CHAINED pipelines.
+- **Raft group.** Each CHAINED pipeline keeps its Raft group.
+  The group provides peer membership, the datastream server and pipeline 
management.
+  It is never used for write log entries.
+- **Durability.** Data and PutBlock are ack'ed on **every** datanode in the 
chain before the client is acknowledged (all-ack).
+- **bcsId.** The block commit sequence ID is always `0` for CHAINED blocks and 
containers.
+
+Branch `master` already has the datanode/client mechanism,
+the **PutBlock-without-Raft** stream path (HDDS-15757/16008/16746/16910/17093).
+This RFC turns it into a first-class replication type,
+with SCM, OM and ReplicationManager support.
+
+### 1.3 Functional Goals
+
+1. **Zero Raft log entries per block.** No entries for writes, PutBlock 
(close, flush or hsync), or stream close.
+2. **Same durability as V1/V2, or better.** An acknowledged PutBlock is ack'ed 
on all replicas, not just a majority.
+3. **hflush/hsync support** through stream commands 
(`DataStreamOutput#commandAsync`).
+4. **A first-class `CHAINED` replication type.**
+   - Supported configs: `CHAINED/ONE` and `CHAINED/THREE`.
+   - Selected through the bucket/key replication config, as with EC.
+   - Accepted by `ozone.replication.allowed-configs`.
+5. **Dedicated CHAINED pipelines and containers.** They are never mixed with 
RATIS pipelines or containers.
+6. **CHAINED containers are never QUASI_CLOSED.** Lifecycle: `OPEN → CLOSING → 
CLOSED`.
+7. **Full ReplicationManager coverage.** Under-, over- and mis-replication 
handling for CHAINED containers,
+   using the existing replicated-container (Ratis-style) handlers.
+8. **Rolling-upgrade safe.**
+   - Version gates: `HDDSVersion` / `OzoneManagerVersion`.
+   - Layout finalization before CHAINED can be used.
+
+### 1.4 Non-Functional Requirements
+
+| NFR | Target |
+|---|---|
+| Write throughput (large sequential keys, CHAINED/THREE vs V2) | **TBD** 
(e.g. ≥ X% improvement; to be set from benchmarks) |
+| PutBlock latency p50/p99 | **TBD**. Expected to drop by roughly one Raft 
commit + apply round trip. |
+| Raft log write bytes per GB written | **0** for writes (control traffic 
only) |
+| Leader CPU | No per-block work on the Raft leader. Work is spread along the 
chain. |
+| RocksDB writes | No more than V2 (one PutBlock per flush/close per replica) |
+| Durability | Ack only after data + PutBlock are synced on all replicas 
(HDDS-17093) |
+| Compatibility | No change to existing RATIS/EC data paths. No new protobuf 
fields. No RocksDB layout change. |
+| Availability | One failed or slow datanode fails the in-flight block. The 
client retries on another pipeline (§4.1.3). |
+
+---
+
+## 2. Out of Scope / Future Works
+
+- **EC with streaming.** EC keys keep the existing direct-write path.
+  EC may switch to `CHAINED/ONE` in the future.
+- **Raft-less pipeline management.** Creating and closing pipelines still uses 
the existing Raft group commands
+  (`CreatePipelineCommand`, `ClosePipelineCommand`).
+  Close pipelines will remove the Raft group.
+- **Migrating existing RATIS keys or containers to CHAINED.** CHAINED applies 
only to new writes.
+- **Replication factors other than ONE and THREE.**
+- **Tolerating a partial ack.** A write that is not acked by every replica is 
a failed write.
+  Majority/quorum acks for streaming are not proposed.
+- **Orphan-block garbage collection.** Blocks written but never committed to 
OM are the same issue as for EC and V2 today,
+  and are handled separately.
+- **Multi-writer appends.** Ozone blocks stay single-writer.
+- **Container layouts other than FILE_PER_BLOCK.** FILE_PER_BLOCK is the only 
supported layout.
+
+---
+
+## 3. Proposed Architecture
+
+### 3.1 Component View
+
+```
+                              ┌──────────────────────┐
+                              │         OM           │
+        ┌────────────────────►│ openKey/allocateBlock│
+        |       allocateBlock │ commitKey / hsync    │
+        |     (CHAINED/THREE) └──────────────────────┘
+        |                                  ▲ pipeline/conatainers
+        |                                  │ (CHAINED/THREE) 
+        |                                  │
+ ┌──────┴───────┐             ┌──────────────────────────────────────┐
+ │   Client     │             │                 SCM                  │
+ │ (OFS / S3G / │             │ PipelineFactory[CHAINED] =           │
+ │  RpcClient)  │             │      ChainedPipelineProvider         │
+ │              │             │ WritableContainerFactory[CHAINED]    │
+ │ KeyDataStream│             │ ReplicationManager (Ratis handlers)  │
+ │  Output      │             │ CloseContainer: force=true(non-RATIS)│
+ │ BlockData-   │             └───────▲────────────────┬─────────────┘
+ │ StreamOutput │                     │ reports        │ Create/ClosePipeline,
+ └──────┬───────┘                     │ (container,    │ CloseContainer(force)
+        │                             │  pipeline)     ▼
+        │ Ratis DataStream   ┌────────┴──────────── CHAINED pipeline 
────────────────┐
+        │ (header, packets,  │                                                 
      │
+        │  commands, CLOSE)  │  ┌────────────┐  stream  ┌────────────┐ stream 
┌────────────┐
+        └───────────────────►│  │ Primary DN │─────────►│    DN2     
│───────►│    DN3     │
+          ◄── all-ack reply ─┤  │ (closest)  │◄─ ack ───│            │◄─ ack 
─│            │
+                             │  └─────┬──────┘          └─────┬──────┘        
└──────┬─────┘
+                             │        │ KeyValueStreamDataChannel 
(FILE_PER_BLOCK)   │
+                             │        ▼ data file + RocksDB PutBlock, synced   
      ▼
+                             │                                                 
      │
+                             │  ┄┄┄┄┄┄┄┄┄┄┄┄ Raft group (control only) 
┄┄┄┄┄┄┄┄┄┄┄┄┄ │
+                             │  membership, datastream server, group 
add/remove;     │
+                             │  NO log entries for WriteChunk / PutBlock / 
close     │
+                             
└───────────────────────────────────────────────────────┘
+```
+
+### 3.2 Write Sequence (one block)
+
+```
+Client                    Primary DN               DN2                 DN3     
      OM
+  │ allocateBlock(CHAINED/THREE) 
───────────────────────────────────────────────────►│
+  │◄───────────────────────────────────────── block + CHAINED pipeline + token 
──────│
+  │ stream(header: StreamInitWithPutBlock, routing table P→2→3)                
      │
+  │──────────────────────────►│──────────────────►│─────────────────►│         
      │
+  │  (each DN: validate token, open KeyValueStreamDataChannel; no 
container2BCSIDMap)│
+  │ writeAsync(data packet)*  │                   │                  │         
      │
+  │──────────────────────────►│──────────────────►│─────────────────►│         
      │
+  │ [hsync] commandAsync(PutBlock)                │                  │         
      │
+  │──────────────────────────►│──────────────────►│─────────────────►│ 
persist+sync  │
+  │◄──────── reply (all peers OK, identical) ─────┴──────────────────┘         
      │
+  │ hsync(key, length) 
─────────────────────────────────────────────────────────────►│
+  │ writeAsync(tail = |data|PutBlock|int len|, CLOSE)                          
      │
+  │──────────────────────────►│──────────────────►│─────────────────►│         
      │
+  │      each DN: closeWithStreamPutBlock → write data, persist PutBlock, sync 
      │
+  │◄──────── reply (all peers OK, identical) ─────┴──────────────────┘         
      │
+  │ commitKey(block, length) 
───────────────────────────────────────────────────────►│
+```
+
+### 3.3 Key Architectural Properties
+
+- **All-ack.** The Ratis datastream server replies to a CLOSE or command only 
after every peer in the routing table succeeds with an identical reply.
+  Ratis performs this check in `DataStreamManagement#checkSuccessRemoteWrite`.
+- **Chain routing.** The primary is `pipeline.getClosestNode()`,
+  and the chain comes from `RatisHelper#getRoutingTable(pipeline)` 
(`ozone.client.datastream.pipeline.mode=true`).
+- **No Raft ordering of writes.** Each block is a separate stream with a 
single writer.
+  Ordering inside a block is the stream's byte order, and there is no 
cross-block ordering.
+- **bcsId = 0.** No Raft index exists to assign. Because every CHAINED replica 
has bcsId 0,
+  replicas cannot disagree on bcsId, which is why QUASI_CLOSED is unnecessary.
+
+---
+
+## 4. Detailed Component Design
+
+### 4.1 Client
+
+#### 4.1.1 Replication selection
+- **Selecting V3.** Users choose V3 through the bucket or key replication 
config:
+  - `ozone sh bucket create --type CHAINED --replication THREE /vol/bucket`
+  - or a key-level `ReplicationConfig`
+- **`ChainedReplicationConfig`** (new, in `hadoop-hdds/common`):
+  - Implements `ReplicatedReplicationConfig` and reuses `ReplicationFactor`, 
like `RatisReplicationConfig`.
+  - Valid instances: `CHAINED/ONE` and `CHAINED/THREE`.
+  - Wired into `ReplicationConfig.fromProto`, `fromProtoTypeAndFactor`, 
`parse` and `getDefault`.
+- **Other client wiring.**
+  - Remove `@Deprecated` from `ReplicationType.CHAINED`.
+  - The shell's `ReplicationOptions` stops rejecting CHAINED.
+
+#### 4.1.2 Stream output
+- **`RpcClient#createDataStreamOutput` / `KeyDataStreamOutput`.**
+  A CHAINED key always uses the stream path, with `putBlockWithoutRaft = true`,
+  whatever `ozone.client.datastream.putblock.without.raft.enabled` says.
+  That flag keeps its V2 meaning for RATIS keys.
+- **`XceiverClientCreator`.** `case CHAINED` returns `XceiverClientRatis`,
+  which provides the DataStream API for the pipeline.
+- **`BlockDataStreamOutput`:**
+  - The header uses `Type.StreamInitWithPutBlock` with 
`WriteChunkRequestProto` (block ID, checksum) and `writePipelineVersion`.
+  - Flush and hsync send `PutBlock` with `DataStreamOutput#commandAsync`.
+  - Close sends the tail `|data|PutBlockRequestProto|int32 length|` with 
`StandardWriteOption.CLOSE`.
+    The PutBlock is at most 1 MB (`PUT_BLOCK_REQUEST_LENGTH_MAX`), a design 
constant.
+  - The existing version gate `allDatanodesSupportStreamPutBlock` 
(`HDDSVersion.STREAM_PUTBLOCK_WITHOUT_RAFT`)
+    is a hard requirement for CHAINED.
+    SCM only builds CHAINED pipelines from datanodes that support it (§4.3.1).
+
+#### 4.1.3 Failure handling and retry
+- **Failed or unmatched reply.** If a CLOSE or command reply is a failure or a 
mismatch, the block write fails.
+- **Retry.** `KeyDataStreamOutput` reacts the same way it does today for V1/V2:
+  - Add the pipeline to the `ExcludeList`; on `CONTAINER_NOT_OPEN` / 
`CLOSED_CONTAINER_IO`, add the container instead.
+  - Allocate a new block from OM and SCM.
+  - Rewrite the buffered, unacked data (`ozone.client.datastream.window.size`).
+- **The abandoned block** keeps the length of its last all-acked PutBlock, if 
there was one;
+  otherwise it is dropped from the key. Data beyond that length is never 
committed (§5.4).
+- **Known issue (prerequisite fix).** In 
`BlockDataStreamOutput#executePutBlock`,
+  a failed CLOSE reply re-calls `executePutBlock` on a stream that is already 
closed.
+  This can recurse without bound. It must be fixed before CHAINED is GA.
+
+### 4.2 Datanode
+
+| Area | Behavior for CHAINED |
+|---|---|
+| Stream init | `ContainerStateMachine#stream` → `HddsDispatcher` (op 
`STREAM_INIT`) validates the block token. Auto-creating the container works as 
for V2. **Change:** for a `StreamInitWithPutBlock` request, the container is 
added to `container2BCSIDMap` only when the group's recorded type is RATIS. 
When the type is CHAINED or unknown, it is not added. Plain `StreamInit` (V2) 
is unchanged. |
+| Data | `KeyValueStreamDataChannel` writes packets to the FILE_PER_BLOCK 
block file. |
+| Flush/hsync | `ContainerStateMachine#streamCommand` persists the PutBlock 
(`ContainerStateMachine#streamPutBlock`) and syncs it before replying. |
+| Close | `KeyValueStreamDataChannel#closeWithStreamPutBlock` parses the tail, 
writes the remaining data, persists the PutBlock with bcsId 0 
(`BlockManagerImpl#persistPutBlock`), and syncs data and metadata before 
replying (HDDS-17093). |
+| Container close | **Change:** `CloseContainerCommandHandler` closes CHAINED 
containers directly (`controller.closeContainer`) even if the pipeline's Raft 
group exists. It never submits CloseContainer through Raft and never 
quasi-closes them. |
+| Raft group removal | `ContainerStateMachine#notifyGroupRemove` only goes 
through `container2BCSIDMap`. CHAINED containers are not in it, so they are 
never quasi-closed. |
+| Restart | `ContainerSet#buildMissingContainerSetAndValidate` checks 
containers against the Raft snapshot's bcsId map. CHAINED containers are not in 
it, so they are not marked UNHEALTHY. The recorded group types are in memory 
only, so every group reloaded at startup has an unknown type. SCM closes the 
restarted datanode's CHAINED pipelines (§4.3.2). |
+| Pipeline creation | **Change:** `CreatePipelineCommandHandler` creates the 
Raft group (`XceiverServerRatis#addGroup`) as today. It also records 
`CreatePipelineCommandProto.type` for the group in `XceiverServerRatis`, on 
every command, including when the group already exists (`isExist`). |
+
+**How the datanode knows a container is CHAINED.** A CHAINED pipeline creates 
only CHAINED containers, so the datanode goes by the pipeline's type.
+- **Recording the type.** `XceiverServerRatis` keeps an in-memory `RaftGroupId 
→ ReplicationType` map, filled from `CreatePipelineCommandProto.type`. It is 
not persisted. Nothing is added to disk and the protobuf doesn't change.
+  - The type must be recorded even when the group already exists. The datanode 
that handles the command first adds the group on its peers through 
`getGroupManagementApi(peer).add(group)`. When a peer's own command arrives, 
`isExist` is already true, and today the handler skips its body.
+  - So the state machine can exist before its type is known. The type is 
looked up at stream init, from the request's group 
(`ContainerStateMachine#getGroupId`), not when the state machine is built.
+- **Stream init.** A `StreamInitWithPutBlock` request adds its container to 
`container2BCSIDMap` only when the group is known to be RATIS. CHAINED and 
unknown groups are treated the same way: the container is not added.
+  - **Unknown type for CHAINED.** This can happen just after pipeline 
creation, or after a restart until SCM's `ClosePipelineCommand` arrives. The 
Ratis and DataStream servers start in `VersionEndpointTask` 
(`ozoneContainer.start`), before the datanode registers. Not adding the 
container is exactly the CHAINED behavior, so this window is harmless.
+  - **Unknown type for RATIS.** This only happens after a restart. A RATIS 
container that is first auto-created by `StreamInitWithPutBlock` 
(`ozone.client.datastream.putblock.without.raft.enabled`) in this state is not 
tracked in `container2BCSIDMap`, so the bcsId check at restart skips it. That 
costs nothing: such writes have no Raft write entries, so the bcsId is 0 anyway.
+- **Close.** SCM sends `force=true` for every non-RATIS container. The 
datanode closes directly when `force` is set and the container is not tracked 
in `container2BCSIDMap`, i.e. it was never written through Raft. This holds 
whether the group's type is known or not. A RATIS container written through 
Raft is always in that map, so a forced close of a RATIS container keeps 
today's behavior.
+
+### 4.3 SCM
+
+#### 4.3.1 `ChainedPipelineProvider` (new)
+- **Why it is needed.**
+  - `PipelineFactory` maps each `ReplicationType` to one `PipelineProvider` 
and registers only STAND_ALONE, RATIS and EC today.
+  - `RatisPipelineProvider` extends `PipelineProvider<RatisReplicationConfig>`
+    and hardcodes `RatisReplicationConfig.getInstance(factor)` in 
`newPipelineBuilder`,
+    so it can only build RATIS pipelines.
+- **What it reuses.** It extends `PipelineProvider<ChainedReplicationConfig>`
+  and reuses the Raft-group logic pulled out of `RatisPipelineProvider` rather 
than copied:
+  - datanode choice: `pickNodesNotUsed` for ONE, `PipelinePlacementPolicy` for 
THREE;
+  - `LeaderChoosePolicy`;
+  - `CreatePipelineCommand(id, CHAINED, factor, dns, suggestedLeader)` to each 
datanode;
+  - `ClosePipelineCommand` on close.
+- **Differences from RATIS:**
+  - **Datastream port.** The `RATIS_DATASTREAM` port is required strictly.
+    RATIS falls back to datanodes without it in `chooseThreeFactorDatanodes`; 
CHAINED does not.
+  - **Datanode version.** Only datanodes at 
`HDDSVersion.STREAM_PUTBLOCK_WITHOUT_RAFT` or later are eligible.
+  - **Pipeline limit.** The per-datanode limit 
(`ozone.scm.datanode.pipeline.limit`) counts **both** RATIS/THREE and 
CHAINED/THREE Raft groups. Today 
`PipelinePlacementPolicy#currentRatisThreePipelineCount` and 
`RatisPipelineProvider#exceedPipelineNumberLimit` count RATIS/THREE only.
+
+#### 4.3.2 Other SCM pipeline changes
+- **`PipelineFactory`** registers `CHAINED → ChainedPipelineProvider`.
+- **`BackgroundPipelineCreator#getReplicationConfigs`** also creates CHAINED 
pipelines
+  when CHAINED is in `ozone.replication.allowed-configs`:
+  - THREE always;
+  - ONE under the existing `ozone.scm.pipeline.creation.auto.factor.one` rule.
+- **`PipelineManagerImpl`:**
+  - `closePipelinesMissingDataStreamPort` includes CHAINED.
+  - `factorOne` handles CHAINED.
+  - Remove `case CHAINED: // Not supported`.
+- **Datanode restart.** When a datanode SCM already knows registers again (the 
`else` branch of `SCMNodeManager#register`),
+  the leader SCM closes every CHAINED pipeline that contains it. That datanode 
has lost its in-memory group types (§4.2).
+  - Today that branch fires no pipeline event. `StaleNodeHandler` closes 
pipelines only after `ozone.scm.stale.node.interval`,
+    which a fast restart never reaches. So this needs a new event and handler.
+  - RATIS pipelines are not closed.
+  - Re-registering without a restart (for example after a `ReregisterCommand`) 
also triggers the close.
+    That is unnecessary, but safe.
+  - If SCM restarts as well, the datanode registers as a new node, so this 
trigger doesn't fire.
+    The datanode stays correct, because unknown-type groups are handled safely 
(§4.2).
+- **`PipelineReportHandler#setPipelineLeaderId`.** A CHAINED/ONE pipeline has 
no leader flag, the same as RATIS/ONE.
+- **Safe mode.** The pipeline rules count RATIS/THREE only, so safe mode is 
unchanged.

Review Comment:
   That's a good point.  Let me think about it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to