yandrey321 opened a new pull request, #11783:
URL: https://github.com/apache/ozone/pull/11783
## What changes were proposed in this pull request?
**Problem.** Since HDDS-16396, positioned reads (preads) of an EC key run
concurrently and leave the sequential
position alone. They do it by building a new reader for each call:
`ECBlockInputStreamProxy.readPositioned` creates
a throwaway proxy, seeks it, reads and closes it. Every pread therefore
opens new per-index block streams. That means
a GetBlock call to a datanode for each index the pread touches, before any
data is read. A 4 KB pread costs a GetBlock
plus a ReadChunk, and a 1 MB pread that spans two cells costs two of each.
**Implementation idea.** A healthy pread reads through the proxy's own
long-lived `ECBlockInputStream`, using the
block streams it already has open:
- `ECBlockInputStream.readPositioned`, which used to throw, splits the range
into cells by data index. It reads each
index's cells with the stateless positioned read of that index's block
stream (`BlockInputStream.readPositioned`,
HDDS-16396). Neither the EC stream nor the per-index streams move.
- With `ozone.client.ec.parallel.read.enabled` (HDDS-16867), the indexes are
read in parallel, as in a sequential
read. Otherwise they are read one after another.
- `ECBlockInputStreamProxy.readPositioned` takes this path while the stream
is healthy. If any index fails, the
failed locations are recorded, and that call alone falls back to the
existing per-call reader, which reconstructs
the data. Once a location has failed, later preads take the reconstruction
path, as they do today.
**Key decisions.**
- **A pread never changes shared state.** A failed index isn't switched to a
spare and its stream isn't closed;
the error goes to the proxy, which reconstructs for that call. Spare
switching stays with the sequential read.
- **Thread safety:**
- A small lock covers the lazy creation of per-index streams, their close,
and the switch to a spare, because
preads open streams without the stream monitor.
- After switching to a spare, the sequential read closes the slot again,
in case a concurrent pread reopened the
stream on the failed location.
- If a sequential read closes a stream that a pread is using, the pread
fails and falls back. That is slower but
still correct.
- **One code path for cell reads.** The cell split, the parallel fan-out and
the failure report are extracted from
the HDDS-16867 parallel read and shared. The per-index read gets a
`positioned` flag.
- **Degraded reads are unchanged.** When a reconstruction reader is in use,
each pread still gets its own decoder
and buffers. Reading only the requested range in that case is a separate
sub-task.
- `ECBlockReconstructedStripeInputStream` keeps rejecting preads, and still
doesn't report `PREADBYTEBUFFER`.
**Side effects.**
- Per-index block streams now serve sequential reads and preads at the same
time. Both use `BlockInputStream`'s
existing concurrency contract, the same one non-EC keys already rely on.
- One stream that fails during a pread costs that call a failed attempt plus
a reconstruction. Before, the per-call
reader switched to a spare first, if one existed.
Generated-by: Claude Code (Claude Opus 5.5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16870
## How was this patch tested?
- **Benchmark:** new `TestEcPositionedReadBenchmark` (`@Tag("benchmark")`,
so it doesn't run in CI).
- Setup: MiniOzone with 5 datanodes, a 48 MB RS(3,2) key with 1 MB cells,
and one open stream. Preads go to random
offsets; 5 rounds after 1 warm-up. The table gives the median throughput
of the rounds and the latency over all
rounds.
- The A/B ran in sequence, base `hdds-client` and then this patch, twice;
the two repetitions agree within 4%. The
table shows repetition 1.
| pread | threads | ops/s before | ops/s after | p50 ms before / after |
p99 ms before / after | GetBlock per pread before / after |
|---|---|---|---|---|---|---|
| 4 KB | 1 | 1604 | 2562 (+60%) | 0.64 / 0.37 | 1.24 / 1.12 | 1.01 / 0.05 |
| 4 KB | 8 | 4436 | 8537 (+92%) | 1.72 / 0.86 | 3.53 / 1.80 | 1.01 / 0.01 |
| 64 KB | 1 | 2087 | 3316 (+59%) | 0.44 / 0.27 | 0.94 / 0.66 | 1.07 / 0.05
|
| 64 KB | 8 | 4206 | 8148 (+94%) | 1.77 / 0.90 | 4.15 / 2.22 | 1.06 / 0.01
|
| 1 MB | 1 | 677 | 904 (+34%) | 1.46 / 1.04 | 4.18 / 1.90 | 2.02 / 0.24 |
| 1 MB | 8 | 2044 | 3139 (+54%) | 3.78 / 2.39 | 7.21 / 5.19 | 2.00 / 0.03 |
- ReadChunk calls per pread are identical in both arms, so the data path
is unchanged; only the GetBlock calls go.
- The GetBlock calls that remain come from opening the stream once per
round, spread over that round's preads.
- Small preads gain the most, because a GetBlock round trip is a large
part of their cost; with 8 threads, their
throughput nearly doubles.
- The 1 MB p99 at 1 thread falls by half because the reads no longer wait
on a GetBlock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
