szetszwo commented on code in PR #11982: URL: https://github.com/apache/ozone/pull/11982#discussion_r4621146556
########## streaming-without-raft.md: ########## @@ -0,0 +1,578 @@ +# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication) + +| Field | Value | +|---------|-------------------------------------------| +| Status | Draft 2026-11-02 | +| Author | Tsz-Wo Nicholas Sze, Rui Wang | +| AI | Claude | +| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093 | + +--- + +## 1. Executive Summary & Goals + +### 1.1 Background + +Apache Ozone currently has three write pipelines: + +| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per block | Replication type | +|---|---|---|---|---| +| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft | 1 per chunk + 1 per PutBlock | `RATIS` | +| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock | `RATIS` | +| **EC** | Direct gRPC writes of data and parity chunks to each datanode | `PutBlock` direct to each datanode | 0 | `EC` | + +In V1 and V2, every write is ordered by the Raft leader and written to the Raft log, which costs: + +- **Raft log I/O.** Each entry is written to the Raft log on every datanode, + in addition to the container data and RocksDB metadata. +- **Leader CPU.** The leader is a hotspot for ordering, replicating and applying log entries. +- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and apply. + +Raft totally orders writes **across blocks in a container**. +Ozone blocks have a single writer and are immutable after commit, so that ordering is not needed for block data. + +### 1.2 Proposal + +**Write Pipeline V3** sends **both StreamData and PutBlock over Ratis DataStream** and writes **no Raft log entries for any write**. +V3 is identified by the previously unused `ReplicationType.CHAINED` (`= 3` in `hdds.proto`): + +- **Keys and containers.** CHAINED keys are written to CHAINED containers. +- **Pipelines.** CHAINED containers live on dedicated CHAINED pipelines. +- **Raft group.** Each CHAINED pipeline keeps its Raft group. + The group provides peer membership, the datastream server and pipeline management. + It is never used for write log entries. +- **Durability.** Data and PutBlock are ack'ed on **every** datanode in the chain before the client is acknowledged (all-ack). +- **bcsId.** The block commit sequence ID is always `0` for CHAINED blocks and containers. Review Comment: See HDDS-17845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
