ivandika3 commented on code in PR #11982:
URL: https://github.com/apache/ozone/pull/11982#discussion_r4623745204
##########
streaming-without-raft.md:
##########
@@ -0,0 +1,578 @@
+# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication)
+
+| Field | Value |
+|---------|-------------------------------------------|
+| Status | Draft 2026-11-02 |
+| Author | Tsz-Wo Nicholas Sze, Rui Wang |
+| AI | Claude |
+| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093 |
+
+---
+
+## 1. Executive Summary & Goals
+
+### 1.1 Background
+
+Apache Ozone currently has three write pipelines:
+
+| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per
block | Replication type |
+|---|---|---|---|---|
+| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft
| 1 per chunk + 1 per PutBlock | `RATIS` |
+| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via
Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock |
`RATIS` |
+| **EC** | Direct gRPC writes of data and parity chunks to each datanode |
`PutBlock` direct to each datanode | 0 | `EC` |
+
+In V1 and V2, every write is ordered by the Raft leader and written to the
Raft log, which costs:
+
+- **Raft log I/O.** Each entry is written to the Raft log on every datanode,
+ in addition to the container data and RocksDB metadata.
+- **Leader CPU.** The leader is a hotspot for ordering, replicating and
applying log entries.
+- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and
apply.
+
+Raft totally orders writes **across blocks in a container**.
+Ozone blocks have a single writer and are immutable after commit, so that
ordering is not needed for block data.
+
+### 1.2 Proposal
+
+**Write Pipeline V3** sends **both StreamData and PutBlock over Ratis
DataStream** and writes **no Raft log entries for any write**.
+V3 is identified by the previously unused `ReplicationType.CHAINED` (`= 3` in
`hdds.proto`):
+
+- **Keys and containers.** CHAINED keys are written to CHAINED containers.
+- **Pipelines.** CHAINED containers live on dedicated CHAINED pipelines.
+- **Raft group.** Each CHAINED pipeline keeps its Raft group.
+ The group provides peer membership, the datastream server and pipeline
management.
+ It is never used for write log entries.
+- **Durability.** Data and PutBlock are ack'ed on **every** datanode in the
chain before the client is acknowledged (all-ack).
+- **bcsId.** The block commit sequence ID is always `0` for CHAINED blocks and
containers.
+
+Branch `master` already has the datanode/client mechanism,
+the **PutBlock-without-Raft** stream path (HDDS-15757/16008/16746/16910/17093).
+This RFC turns it into a first-class replication type,
+with SCM, OM and ReplicationManager support.
+
+### 1.3 Functional Goals
+
+1. **Zero Raft log entries per block.** No entries for writes, PutBlock
(close, flush or hsync), or stream close.
+2. **Same durability as V1/V2, or better.** An acknowledged PutBlock is ack'ed
on all replicas, not just a majority.
+3. **hflush/hsync support** through stream commands
(`DataStreamOutput#commandAsync`).
+4. **A first-class `CHAINED` replication type.**
+ - Supported configs: `CHAINED/ONE` and `CHAINED/THREE`.
+ - Selected through the bucket/key replication config, as with EC.
+ - Accepted by `ozone.replication.allowed-configs`.
+5. **Dedicated CHAINED pipelines and containers.** They are never mixed with
RATIS pipelines or containers.
+6. **CHAINED containers are never QUASI_CLOSED.** Lifecycle: `OPEN → CLOSING →
CLOSED`.
+7. **Full ReplicationManager coverage.** Under-, over- and mis-replication
handling for CHAINED containers,
+ using the existing replicated-container (Ratis-style) handlers.
+8. **Rolling-upgrade safe.**
+ - Version gates: `HDDSVersion` / `OzoneManagerVersion`.
+ - Layout finalization before CHAINED can be used.
+
+### 1.4 Non-Functional Requirements
+
+| NFR | Target |
+|---|---|
+| Write throughput (large sequential keys, CHAINED/THREE vs V2) | **TBD**
(e.g. ≥ X% improvement; to be set from benchmarks) |
+| PutBlock latency p50/p99 | **TBD**. Expected to drop by roughly one Raft
commit + apply round trip. |
+| Raft log write bytes per GB written | **0** for writes (control traffic
only) |
+| Leader CPU | No per-block work on the Raft leader. Work is spread along the
chain. |
+| RocksDB writes | No more than V2 (one PutBlock per flush/close per replica) |
+| Durability | Ack only after data + PutBlock are synced on all replicas
(HDDS-17093) |
+| Compatibility | No change to existing RATIS/EC data paths. No new protobuf
fields. No RocksDB layout change. |
+| Availability | One failed or slow datanode fails the in-flight block. The
client retries on another pipeline (§4.1.3). |
+
+---
+
+## 2. Out of Scope / Future Works
+
+- **EC with streaming.** EC keys keep the existing direct-write path.
+ EC may switch to `CHAINED/ONE` in the future.
+- **Raft-less pipeline management.** Creating and closing pipelines still uses
the existing Raft group commands
+ (`CreatePipelineCommand`, `ClosePipelineCommand`).
+ Close pipelines will remove the Raft group.
+- **Migrating existing RATIS keys or containers to CHAINED.** CHAINED applies
only to new writes.
+- **Replication factors other than ONE and THREE.**
+- **Tolerating a partial ack.** A write that is not acked by every replica is
a failed write.
+ Majority/quorum acks for streaming are not proposed.
+- **Orphan-block garbage collection.** Blocks written but never committed to
OM are the same issue as for EC and V2 today,
+ and are handled separately.
+- **Multi-writer appends.** Ozone blocks stay single-writer.
+- **Container layouts other than FILE_PER_BLOCK.** FILE_PER_BLOCK is the only
supported layout.
+
+---
+
+## 3. Proposed Architecture
+
+### 3.1 Component View
+
+```
+ ┌──────────────────────┐
+ │ OM │
+ ┌────────────────────►│ openKey/allocateBlock│
+ | allocateBlock │ commitKey / hsync │
+ | (CHAINED/THREE) └──────────────────────┘
+ | ▲ pipeline/conatainers
+ | │ (CHAINED/THREE)
+ | │
+ ┌──────┴───────┐ ┌──────────────────────────────────────┐
+ │ Client │ │ SCM │
+ │ (OFS / S3G / │ │ PipelineFactory[CHAINED] = │
+ │ RpcClient) │ │ ChainedPipelineProvider │
+ │ │ │ WritableContainerFactory[CHAINED] │
+ │ KeyDataStream│ │ ReplicationManager (Ratis handlers) │
+ │ Output │ │ CloseContainer: force=true(non-RATIS)│
+ │ BlockData- │ └───────▲────────────────┬─────────────┘
+ │ StreamOutput │ │ reports │ Create/ClosePipeline,
+ └──────┬───────┘ │ (container, │ CloseContainer(force)
+ │ │ pipeline) ▼
+ │ Ratis DataStream ┌────────┴──────────── CHAINED pipeline
────────────────┐
+ │ (header, packets, │
│
+ │ commands, CLOSE) │ ┌────────────┐ stream ┌────────────┐ stream
┌────────────┐
+ └───────────────────►│ │ Primary DN │─────────►│ DN2
│───────►│ DN3 │
Review Comment:
AFAIK, Ratis allows both star topology and chained topology? Both have their
pros and cons. Star topology supposedly improves latency since it reduce the
length of the pipeline, but requires 2x network throughput from the first node
which can make the network bandwidth to be skewed on the first node which can
also affect latency if it causes network issues (e.g. packet loss which
triggers TCP retranmission). Chain topology has higher supposed latency due to
the length of the pipeline, but there won't be asymmetry in the network
bandwidth (each node has the equal network bandwidth which is easier to reason
about). From consistency point of view, the chained topology is not strictly
needed since Ozone write pipeline does not guarantee global strong consistency
like CRAQ / Chain replication and also Ozone write pipeline is append only so
using ackedLength should be enough.
One way to find out is to benchmark.
> we can also send back ACK to the client as soon as we get write quorum,
so we dont need to wait for the last write to complete.
This is durability related issue, I prefer if we safely persist all the
blocks in all nodes in the chain before acknowledging to the client and
committing the key to OM. The existing Ratis pipeline MAJORITY_COMMITTED
guarantee is very weak since it only guarantees that block has been persisted
in the leader only so a single machine failure for the leader can trigger data
loss where the client believes that it has committed key up and causes
BCSID_MISMATCH. I believe from user-perspective if I see that my data is
guaranteed to be 3-replicated it's that it has actually been persisted to 3
datanodes not just "promised" to be persisted (like in Raft logs that are
committed but not applied yet).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]