[
https://issues.apache.org/jira/browse/HDDS-17036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17036:
-------------------------------------
Assignee: Andrey Yarovoy
> Pipeline EC stripe writes: several stripes in flight per key, without a
> per-stripe WriteChunk barrier
> -----------------------------------------------------------------------------------------------------
>
> Key: HDDS-17036
> URL: https://issues.apache.org/jira/browse/HDDS-17036
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> **Problem**
> The EC client writes one stripe at a time. Before moving to the next stripe,
> it waits for every WriteChunk ack, and
> with J2 for PutBlock only every N stripes. The stripe queue
> (`ozone.client.ec.stripe.queue.size`) only buffers
> encoded stripes; it doesn't overlap their transfer. So a single key's
> throughput is bounded by stripe size divided
> by the slowest datanode's round trip plus service time. Raising the queue
> size doesn't help: going from 2 to 8
> only moved work into `close()`.
> **Evidence** (WriteChunk-ack wait, share of flush-thread busy time)
> | Environment | WriteChunk-ack wait |
> |---|---|
> | Real cluster, RTT 0.05 ms, TLS | 31% |
> | Local mini-cluster | 33% |
> That is about 1.6 ms per stripe on the real cluster, even at 0.05 ms RTT. The
> wait grows with RTT: one round trip
> per stripe for WriteChunk, plus PutBlock.
> **Proposal**
> 1. Allow up to W stripes in flight per key, with a new setting. W = 1 is
> today's behaviour.
> 2. Keep every datanode receiving its cells in order.
> `FilePerBlockStrategy.writeChunk` rejects a non-overwrite
> chunk whose offset doesn't equal the current block file length
> (`ChunkUtils.validateChunkSize`,
> `CHUNK_FILE_INCONSISTENCY`).
> - Today `XceiverClientGrpc.sendCommandAsync` opens a new gRPC stream per
> request, so two in-flight requests to
> the same datanode can be handled out of order.
> - Use **one long-lived bidirectional gRPC stream per (block, datanode)** for
> the key's WriteChunks. gRPC
> delivers messages on one stream in order, and `GrpcXceiverService` processes
> a stream's requests in order.
> 3. Track acks per stripe, and release a stripe's buffers when all its cells
> are acknowledged and it is covered by a PutBlock
> 4. On failure, rewrite from the last successful PutBlock, reusing J2's logic.
> That bounds buffered data to about
> W + N stripes per key.
> **Projected effect**
> - Removes WriteChunk round trips from the per-stripe critical path. The flush
> thread then is limited by its own CPU work , and the writer thread by
> encoding.
> - Larger gains at higher RTT, for example across racks or zones.
> **Compatibility**
> - Client-only.
> - The long-lived stream uses the existing `XceiverClientProtocolService`
> bidirectional `send` RPC, so no wire
> change. Verify against old datanodes.
> **Testing**
> - Ordering: unit and integration tests with W > 1 that must never trigger
> `CHUNK_FILE_INCONSISTENCY`.
> - Failure injection at every stage: WriteChunk failure in stripe k while k+1
> is in flight; PutBlock failure;
> datanode restart.
> - Read-back verification.
> - Benchmarks: `TestEcWriteBenchmark` W-arm, freon at 1/4/16 threads, and with
> added network delay (netem) to confirm the RTT sensitivity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
