[
https://issues.apache.org/jira/browse/HDDS-17034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17034:
-------------------------------------
Assignee: Andrey Yarovoy
> Send an EC stripe's cells to their datanodes in parallel instead of serially
> on one flush thread
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-17034
> URL: https://issues.apache.org/jira/browse/HDDS-17034
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> **Problem**
> For each stripe, a single flush thread per key
> (`ECKeyOutputStream.flushStripeToDatanodes` → `writeDataCells` /
> `writeParityCells`) processes the d+p cells one after another. For each cell
> it:
> 1. computes the chunk checksum
> 2. builds the `WriteChunk` request
> 3. hands it to gRPC (`XceiverClientGrpc.sendCommandAsync`)
> That third step copies the 1 MB cell into gRPC's frame buffer
> (`MessageFramer.writeRaw`). grpc-java's send path
> always copies the message into its own buffers, so this copy cannot be
> removed on the client, only taken off the
> critical path.
> **Evidence** (single rs-3-2-1024k writer on the TLS cluster, flush-thread
> busy time)
> | Activity | Share |
> |---|---|
> | WriteChunk serialization | 15.1% |
> | CRC (1 MB checksum unit) | 4.0% |
> | send | 2.2% |
> In total, about 21% of the flush thread's busy time is per-cell CPU work done
> serially.
> - 87% of the serialization samples are the payload memcpy in
> `MessageFramer.writeRaw` (`jbyte_disjoint_arraycopy`).
> - With the default 16 KB checksum unit, the CRC share is larger: 16% of flush
> time in the mini-cluster profile.
> **Proposal**
> Dispatch each cell's checksum, request build and send on a per-stream
> executor, so the d+p cells of a stripe are
> prepared concurrently. The existing barrier stays: the stripe still completes
> when all WriteChunks are
> acknowledged. Per-datanode order is unchanged, because each datanode gets one
> cell per stripe.
> - Executor size: bounded per client, shared across streams, to avoid thread
> explosion with many writers.
> - The flush thread keeps orchestrating: it waits for all cells to be sent and
> acknowledged, then sends PutBlock as
> today.
> **Projected effect**
> Removes most of the ~21% per-cell work from the flush thread's critical path
> for one writer.
> **Compatibility**
> Client-only. No wire, format or configuration change, except an optional
> executor-size setting.
> **Testing**
> - `TestECKeyOutputStream` and the failure-handling tests (a datanode failing
> mid-stripe must still exclude the
> datanode and rewrite the stripe).
> - A test that checksums and chunk metadata are identical to today's.
> - Before and after: freon at 1 thread. Benefit is expected only when the
> flush thread is the bottleneck.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
