Andrey Yarovoy created HDDS-17036:
-------------------------------------
Summary: Pipeline EC stripe writes: several stripes in flight per
key, without a per-stripe WriteChunk barrier
Key: HDDS-17036
URL: https://issues.apache.org/jira/browse/HDDS-17036
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
**Problem**
The EC client writes one stripe at a time. Before moving to the next stripe, it
waits for every WriteChunk ack, and
with J2 for PutBlock only every N stripes. The stripe queue
(`ozone.client.ec.stripe.queue.size`) only buffers
encoded stripes; it doesn't overlap their transfer. So a single key's
throughput is bounded by stripe size divided
by the slowest datanode's round trip plus service time. Raising the queue size
doesn't help: going from 2 to 8
only moved work into `close()`.
**Evidence** (WriteChunk-ack wait, share of flush-thread busy time)
| Environment | WriteChunk-ack wait |
|---|---|
| Real cluster, RTT 0.05 ms, TLS | 31% |
| Local mini-cluster | 33% |
That is about 1.6 ms per stripe on the real cluster, even at 0.05 ms RTT. The
wait grows with RTT: one round trip
per stripe for WriteChunk, plus PutBlock.
**Proposal**
1. Allow up to W stripes in flight per key, with a new setting. W = 1 is
today's behaviour.
2. Keep every datanode receiving its cells in order.
`FilePerBlockStrategy.writeChunk` rejects a non-overwrite
chunk whose offset doesn't equal the current block file length
(`ChunkUtils.validateChunkSize`,
`CHUNK_FILE_INCONSISTENCY`).
- Today `XceiverClientGrpc.sendCommandAsync` opens a new gRPC stream per
request, so two in-flight requests to
the same datanode can be handled out of order.
- Use **one long-lived bidirectional gRPC stream per (block, datanode)** for
the key's WriteChunks. gRPC
delivers messages on one stream in order, and `GrpcXceiverService` processes a
stream's requests in order.
3. Track acks per stripe, and release a stripe's buffers when all its cells are
acknowledged and it is covered by a PutBlock
4. On failure, rewrite from the last successful PutBlock, reusing J2's logic.
That bounds buffered data to about
W + N stripes per key.
**Projected effect**
- Removes WriteChunk round trips from the per-stripe critical path. The flush
thread then is limited by its own CPU work , and the writer thread by encoding.
- Larger gains at higher RTT, for example across racks or zones.
**Compatibility**
- Client-only.
- The long-lived stream uses the existing `XceiverClientProtocolService`
bidirectional `send` RPC, so no wire
change. Verify against old datanodes.
**Testing**
- Ordering: unit and integration tests with W > 1 that must never trigger
`CHUNK_FILE_INCONSISTENCY`.
- Failure injection at every stage: WriteChunk failure in stripe k while k+1 is
in flight; PutBlock failure;
datanode restart.
- Read-back verification.
- Benchmarks: `TestEcWriteBenchmark` W-arm, freon at 1/4/16 threads, and with
added network delay (netem) to confirm the RTT sensitivity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
