Siyao Meng created HDDS-17412:
---------------------------------
Summary: EC positioned read creates a new reader per call, cutting
random pread throughput by about 45%
Key: HDDS-17412
URL: https://issues.apache.org/jira/browse/HDDS-17412
Project: Apache Ozone
Issue Type: Bug
Components: EC Client
Reporter: Siyao Meng
h2. Problem
Since HDDS-16396, {{ECBlockInputStreamProxy#readPositioned}} creates a new EC
reader for every positioned read and closes it afterwards. The cached per-index
{{BlockInputStream}}s are thrown away each time, so every pread issues a fresh
{{GetBlock}} RPC to the datanode for each EC cell it touches before reading any
data.
Compared with its parent commit on a five-datanode mini-cluster (RS-3-2, 1 MiB
cells, single-threaded client):
||EC workload||Parent (reads/s)||HDDS-16396 (reads/s)||Change||
|Random pread 4 KiB|4,926|2,661|-46%|
|Random pread 64 KiB|3,779|2,192|-42%|
|Sequential read 64 KiB|4,023|4,019|-0.1%|
For 2,000 random 64 KiB preads after warm-up, {{GetBlock}} RPCs went from 0 to
2,121 while {{ReadChunk}} stayed at 2,121. Ratis keys are not affected. Reusing
the existing reader for preads (seek, read, restore) restores the parent's
throughput and removes the extra RPCs, which confirms that creating a reader
per call causes the slowdown. Current master has the same behavior.
h2. Proposed fix
Add a fast path for healthy preads in {{ECBlockInputStream}}: map the requested
range to EC cells and call the existing stateless {{readPositioned}} on the
cached per-index block streams, without moving the sequential cursor. Lazy
stream creation stays synchronized; datanode I/O does not hold the lock, so
concurrent preads still overlap. The existing parallel cell read option also
applies to preads.
{{ECBlockInputStreamProxy}} uses this fast path while the block is healthy. On
failure it restores the destination buffer position and falls back to the
existing request-owned reader. That reader starts with the failed indexes
excluded, so it goes straight to a spare replica or reconstruction.
The fix needs no new configuration, wire change or reader pool.
With the fix on current master, same benchmark:
||EC workload||Unpatched (reads/s)||Fixed (reads/s)||
|Random pread 4 KiB|2,776|5,001|
|Random pread 64 KiB|2,283|3,828|
|Positioned scan 64 KiB|2,448|4,074|
Warmed EC {{GetBlock}} RPCs drop to zero. Ratis throughput and EC sequential
reads are unchanged.
h2. Tests
Extend {{TestECBlockInputStream}} and {{TestECBlockInputStreamProxy}} to cover:
* reader reuse: 100 preads use one EC reader (without the fix, 101 are created);
* overlapping concurrent preads;
* cell and stripe boundaries, partial last stripes, EOF, and an unchanged
cursor;
* failure after partial output, retry on a spare replica, reconstruction
fallback, and simultaneous failures on several indexes.
Related: HDDS-16870 proposes stateless positioned reads for EC keys more
broadly.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
