szetszwo commented on code in PR #11982:
URL: https://github.com/apache/ozone/pull/11982#discussion_r4621139158
##########
streaming-without-raft.md:
##########
@@ -0,0 +1,578 @@
+# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication)
+
+| Field | Value |
+|---------|-------------------------------------------|
+| Status | Draft 2026-11-02 |
+| Author | Tsz-Wo Nicholas Sze, Rui Wang |
+| AI | Claude |
+| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093 |
+
+---
+
+## 1. Executive Summary & Goals
+
+### 1.1 Background
+
+Apache Ozone currently has three write pipelines:
+
+| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per
block | Replication type |
+|---|---|---|---|---|
+| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft
| 1 per chunk + 1 per PutBlock | `RATIS` |
+| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via
Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock |
`RATIS` |
+| **EC** | Direct gRPC writes of data and parity chunks to each datanode |
`PutBlock` direct to each datanode | 0 | `EC` |
+
+In V1 and V2, every write is ordered by the Raft leader and written to the
Raft log, which costs:
+
+- **Raft log I/O.** Each entry is written to the Raft log on every datanode,
+ in addition to the container data and RocksDB metadata.
+- **Leader CPU.** The leader is a hotspot for ordering, replicating and
applying log entries.
+- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and
apply.
+
+Raft totally orders writes **across blocks in a container**.
+Ozone blocks have a single writer and are immutable after commit, so that
ordering is not needed for block data.
+
+### 1.2 Proposal
+
+**Write Pipeline V3** sends **both StreamData and PutBlock over Ratis
DataStream** and writes **no Raft log entries for any write**.
+V3 is identified by the previously unused `ReplicationType.CHAINED` (`= 3` in
`hdds.proto`):
+
+- **Keys and containers.** CHAINED keys are written to CHAINED containers.
+- **Pipelines.** CHAINED containers live on dedicated CHAINED pipelines.
+- **Raft group.** Each CHAINED pipeline keeps its Raft group.
+ The group provides peer membership, the datastream server and pipeline
management.
+ It is never used for write log entries.
+- **Durability.** Data and PutBlock are ack'ed on **every** datanode in the
chain before the client is acknowledged (all-ack).
+- **bcsId.** The block commit sequence ID is always `0` for CHAINED blocks and
containers.
+
+Branch `master` already has the datanode/client mechanism,
+the **PutBlock-without-Raft** stream path (HDDS-15757/16008/16746/16910/17093).
+This RFC turns it into a first-class replication type,
+with SCM, OM and ReplicationManager support.
+
+### 1.3 Functional Goals
+
+1. **Zero Raft log entries per block.** No entries for writes, PutBlock
(close, flush or hsync), or stream close.
+2. **Same durability as V1/V2, or better.** An acknowledged PutBlock is ack'ed
on all replicas, not just a majority.
+3. **hflush/hsync support** through stream commands
(`DataStreamOutput#commandAsync`).
+4. **A first-class `CHAINED` replication type.**
+ - Supported configs: `CHAINED/ONE` and `CHAINED/THREE`.
+ - Selected through the bucket/key replication config, as with EC.
+ - Accepted by `ozone.replication.allowed-configs`.
+5. **Dedicated CHAINED pipelines and containers.** They are never mixed with
RATIS pipelines or containers.
+6. **CHAINED containers are never QUASI_CLOSED.** Lifecycle: `OPEN → CLOSING →
CLOSED`.
+7. **Full ReplicationManager coverage.** Under-, over- and mis-replication
handling for CHAINED containers,
+ using the existing replicated-container (Ratis-style) handlers.
+8. **Rolling-upgrade safe.**
+ - Version gates: `HDDSVersion` / `OzoneManagerVersion`.
+ - Layout finalization before CHAINED can be used.
+
+### 1.4 Non-Functional Requirements
+
+| NFR | Target |
+|---|---|
+| Write throughput (large sequential keys, CHAINED/THREE vs V2) | **TBD**
(e.g. ≥ X% improvement; to be set from benchmarks) |
+| PutBlock latency p50/p99 | **TBD**. Expected to drop by roughly one Raft
commit + apply round trip. |
+| Raft log write bytes per GB written | **0** for writes (control traffic
only) |
+| Leader CPU | No per-block work on the Raft leader. Work is spread along the
chain. |
+| RocksDB writes | No more than V2 (one PutBlock per flush/close per replica) |
+| Durability | Ack only after data + PutBlock are synced on all replicas
(HDDS-17093) |
+| Compatibility | No change to existing RATIS/EC data paths. No new protobuf
fields. No RocksDB layout change. |
+| Availability | One failed or slow datanode fails the in-flight block. The
client retries on another pipeline (§4.1.3). |
+
+---
+
+## 2. Out of Scope / Future Works
+
+- **EC with streaming.** EC keys keep the existing direct-write path.
+ EC may switch to `CHAINED/ONE` in the future.
+- **Raft-less pipeline management.** Creating and closing pipelines still uses
the existing Raft group commands
+ (`CreatePipelineCommand`, `ClosePipelineCommand`).
+ Close pipelines will remove the Raft group.
+- **Migrating existing RATIS keys or containers to CHAINED.** CHAINED applies
only to new writes.
+- **Replication factors other than ONE and THREE.**
+- **Tolerating a partial ack.** A write that is not acked by every replica is
a failed write.
+ Majority/quorum acks for streaming are not proposed.
+- **Orphan-block garbage collection.** Blocks written but never committed to
OM are the same issue as for EC and V2 today,
+ and are handled separately.
+- **Multi-writer appends.** Ozone blocks stay single-writer.
+- **Container layouts other than FILE_PER_BLOCK.** FILE_PER_BLOCK is the only
supported layout.
+
+---
+
+## 3. Proposed Architecture
+
+### 3.1 Component View
+
+```
+ ┌──────────────────────┐
+ │ OM │
+ ┌────────────────────►│ openKey/allocateBlock│
+ | allocateBlock │ commitKey / hsync │
+ | (CHAINED/THREE) └──────────────────────┘
+ | ▲ pipeline/conatainers
+ | │ (CHAINED/THREE)
+ | │
+ ┌──────┴───────┐ ┌──────────────────────────────────────┐
+ │ Client │ │ SCM │
+ │ (OFS / S3G / │ │ PipelineFactory[CHAINED] = │
+ │ RpcClient) │ │ ChainedPipelineProvider │
+ │ │ │ WritableContainerFactory[CHAINED] │
+ │ KeyDataStream│ │ ReplicationManager (Ratis handlers) │
+ │ Output │ │ CloseContainer: force=true(non-RATIS)│
+ │ BlockData- │ └───────▲────────────────┬─────────────┘
+ │ StreamOutput │ │ reports │ Create/ClosePipeline,
+ └──────┬───────┘ │ (container, │ CloseContainer(force)
+ │ │ pipeline) ▼
+ │ Ratis DataStream ┌────────┴──────────── CHAINED pipeline
────────────────┐
+ │ (header, packets, │
│
+ │ commands, CLOSE) │ ┌────────────┐ stream ┌────────────┐ stream
┌────────────┐
+ └───────────────────►│ │ Primary DN │─────────►│ DN2
│───────►│ DN3 │
+ ◄── all-ack reply ─┤ │ (closest) │◄─ ack ───│ │◄─ ack
─│ │
+ │ └─────┬──────┘ └─────┬──────┘
└──────┬─────┘
+ │ │ KeyValueStreamDataChannel
(FILE_PER_BLOCK) │
+ │ ▼ data file + RocksDB PutBlock, synced
▼
+ │
│
+ │ ┄┄┄┄┄┄┄┄┄┄┄┄ Raft group (control only)
┄┄┄┄┄┄┄┄┄┄┄┄┄ │
+ │ membership, datastream server, group
add/remove; │
+ │ NO log entries for WriteChunk / PutBlock /
close │
+
└───────────────────────────────────────────────────────┘
+```
+
+### 3.2 Write Sequence (one block)
+
+```
+Client Primary DN DN2 DN3
OM
+ │ allocateBlock(CHAINED/THREE)
───────────────────────────────────────────────────►│
+ │◄───────────────────────────────────────── block + CHAINED pipeline + token
──────│
+ │ stream(header: StreamInitWithPutBlock, routing table P→2→3)
│
+ │──────────────────────────►│──────────────────►│─────────────────►│
│
+ │ (each DN: validate token, open KeyValueStreamDataChannel; no
container2BCSIDMap)│
+ │ writeAsync(data packet)* │ │ │
│
+ │──────────────────────────►│──────────────────►│─────────────────►│
│
+ │ [hsync] commandAsync(PutBlock) │ │
│
+ │──────────────────────────►│──────────────────►│─────────────────►│
persist+sync │
+ │◄──────── reply (all peers OK, identical) ─────┴──────────────────┘
│
+ │ hsync(key, length)
─────────────────────────────────────────────────────────────►│
+ │ writeAsync(tail = |data|PutBlock|int len|, CLOSE)
│
+ │──────────────────────────►│──────────────────►│─────────────────►│
│
+ │ each DN: closeWithStreamPutBlock → write data, persist PutBlock, sync
│
+ │◄──────── reply (all peers OK, identical) ─────┴──────────────────┘
│
+ │ commitKey(block, length)
───────────────────────────────────────────────────────►│
+```
+
+### 3.3 Key Architectural Properties
+
+- **All-ack.** The Ratis datastream server replies to a CLOSE or command only
after every peer in the routing table succeeds with an identical reply.
+ Ratis performs this check in `DataStreamManagement#checkSuccessRemoteWrite`.
+- **Chain routing.** The primary is `pipeline.getClosestNode()`,
+ and the chain comes from `RatisHelper#getRoutingTable(pipeline)`
(`ozone.client.datastream.pipeline.mode=true`).
+- **No Raft ordering of writes.** Each block is a separate stream with a
single writer.
+ Ordering inside a block is the stream's byte order, and there is no
cross-block ordering.
+- **bcsId = 0.** No Raft index exists to assign. Because every CHAINED replica
has bcsId 0,
+ replicas cannot disagree on bcsId, which is why QUASI_CLOSED is unnecessary.
+
+---
+
+## 4. Detailed Component Design
+
+### 4.1 Client
+
+#### 4.1.1 Replication selection
+- **Selecting V3.** Users choose V3 through the bucket or key replication
config:
Review Comment:
Revised the doc. Please check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]