[ 
https://issues.apache.org/jira/browse/HDDS-17752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17752:
----------------------------------
    Description: 
h3. Problem

With streaming read on (\{{ozone.client.stream.readblock.enable=true}}), 
\{{StreamBlockInputStream.readPositioned}} serves every positioned read with 
its own one-shot streaming call. The call never touches the sequential reader 
or its pre-read buffer:

{code:java}
final XceiverClientGrpc client = acquireGrpcClientForReadData(factory, 
pipeline);
final StreamingReader reader = new StreamingReader(client);
client.initStreamRead(blockID, reader, failedStreamingDatanodes); // new gRPC 
call + semaphore permit
try {
return preadOnce(client, reader, pipeline, blockOffset, length, dst); // one 
ReadBlock request, poll until length
} finally {
closeStream(reader); // half-close, release permit
}
{code}

So each pread pays a full call lifecycle, which turns a streaming protocol into 
a unary one with extra steps:
* a client acquire, including the per-call user lookup in 
\{{XceiverClientManager.getPipelineCacheKey}} (HDDS-17716);
* \{{initStreamRead}}: semaphore permit, a new HTTP/2 stream, HEADERS, the 
interceptor chain;
* \{{StreamingReadResponse.awaitReady}}, a wait until the new stream is 
writable;
* on the datanode, a new \{{ServerCall}} and stream observer, plus a block-file 
open (\{{getChunkFile}} stat and open);
* \{{closeStream}}: half-close, trailers, \{{completeStreamRead}}, and on the 
datanode the block file is closed again.

HBase issues most of its reads as preads (\{{BlockIOUtils.preadWithExtra}}), 
including every read of a newly reopened store file, so this per-pread setup 
dominates its streaming read path.

h3. Evidence

HBase YCSB on EC rs-3-2, streaming read on. RegionServer async-profiler, 6 x 60 
s:
* *Wall:* streaming ReadBlock waits are 60.7% of busy RPC handler time, and 
about half of that is preads (\{{preadOnce}} -> \{{StreamingReader.poll}}). 
\{{awaitReady}} alone is about 4% of the streaming wait.
* *CPU, EC read path* (\{{ECBlockInputStream}} read = 100%): 
\{{readPositioned}} is 66%. Inside it, \{{initStreamRead}} is 15%, 
\{{closeStream}} 4.4%, and the \{{streamRead}} send 16% (5.5% of it in 
\{{awaitReady}}).
* *Netty* threads are 47% of RegionServer CPU in this mode: stream creation, 
HEADERS, and a wake-up per call.

h3. What the related changes already cover

* *HDDS-16396* (merged) made EC preads concurrent.
* *HDDS-16870* (merged) makes healthy EC preads reuse the long-lived per-index 
block streams instead of per-call clones. That removes the per-pread GetBlock, 
and it means one \{{StreamBlockInputStream}} per EC index now sees every pread 
on its block. It does not change what that stream does per pread: 
\{{readPositioned}} still opens a one-shot gRPC call. HDDS-16870 reuses the 
Java stream object; this Jira reuses the gRPC stream underneath it. With both, 
an EC pread on a healthy key does no GetBlock and opens no new gRPC call.
* *HDDS-17561* (Patch Available) pools single-request streams for blocking 
\{{sendCommand}} calls (ReadChunk and GetBlock on the classic, non-streaming 
read path) and delivers the reply on the waiting thread. It does not touch 
\{{initStreamRead}} or \{{streamRead}}. Its per-channel pool also cannot hold 
ReadBlock streams as they are, because a datanode binds a ReadBlock stream to 
one block (see below). This Jira borrows its lifecycle rules (drop on error or 
abandon, idle trim, client-side deadline) but needs its own per-block pool.
* *HDDS-17399* (lock-free positioned reads) sets three constraints that this 
design follows: the pool takes no stream monitor, \{{closed}} is volatile and 
re-checked after a reader is pooled, and an idle reader holds no client permit.

h3. Proposed fix

Keep a small pool of idle pread readers in each \{{StreamBlockInputStream}}. 
After a pread that read everything it asked for, its stream stays open; the 
next pread on the block takes the idle reader and sends one more \{{ReadBlock}} 
on it.

* *One reader per pread at a time.* Concurrent preads use different readers. 
This matches the datanode, which serves a stream sequentially.
* *Reuse only after a complete, clean read.* A reader goes back to the pool 
only if it got the full length and its stream is still open with nothing 
unread. The datanode's \{{BlockReadCursor}} sends contiguous ranges ending with 
the one that covers the requested end, so a complete pread leaves nothing in 
flight. Errors, short reads, EOF and \{{OUT_OF_RANGE}} close the reader. 
\{{preadOnce}} still checks each response's offset as a guard.
* *Idle readers hold no client permit.* The permit is returned when a reader 
goes idle and taken again on lease (\{{XceiverClientGrpc.resumeStreamRead}}). 
Otherwise idle readers would eat into the 32 permits per client.
* *Lock-free pool:* a \{{ConcurrentLinkedDeque}} used most-recently-used first, 
with a CAS-bounded count.
* *Close, unbuffer and refresh close all idle readers.* A pool generation 
advances on \{{unbuffer()}} and block-info refresh. A pread that started in an 
older generation, or that finishes after \{{close()}}, closes its reader 
instead of pooling it, so no reader of an old pipeline survives a refresh.
* *A failed reused stream does not blame the datanode.* An idle stream may have 
broken while pooled. Its first failure is retried once on a new stream without 
adding the datanode to \{{failedStreamingDatanodes}}, and the other idle 
readers are dropped. Timeouts and interrupts are not retried, so a hung 
datanode still fails over after one read timeout.
* *Bounded and time-limited*, behind a flag:

||Config||Default||Meaning||
|ozone.client.stream.read.pread.reuse.enabled|false|turn the reuse on|
|ozone.client.stream.read.pread.max-idle-readers|2|idle readers kept per block 
stream|
|ozone.client.stream.read.pread.idle-timeout|1s|close idle readers after this, 
well below the datanode's 1 min stream file idle timeout|

h3. Constraint: one block per stream

The datanode opens the per-stream block file on the first \{{ReadBlock}} and 
reuses it without checking the block ID of later requests 
(\{{KeyValueHandler.readBlockImpl}} opens \{{blockFile}} only if it is not open 
yet). So a reader cannot be shared across blocks or across 
\{{StreamBlockInputStream}} instances, and the pool lives in the block stream, 
not in \{{XceiverClientGrpc}}.

h3. Rejected alternatives

* *Serve preads from the sequential reader.* Its queue holds pre-read data for 
the cursor position, so interleaved pread responses would mean draining or 
discarding that window.
* *Multiplex concurrent preads on one stream.* The datanode serves a stream 
sequentially, and responses carry offsets but no request id.
* *Unary ReadChunk for small preads.* With HDDS-17561 a ReadChunk no longer 
opens a gRPC stream, but it needs the GetBlock chunk list, which the streaming 
path otherwise avoids, and it mixes two read protocols on one block stream.

h3. Results

A/B with only the flag toggled, same build. Secure compose cluster (Kerberos, 
gRPC TLS, 5 datanodes), \{{ozone freon ocpr}} on 48 MB keys, 3 repetitions per 
arm, 108 runs, 0 failures:

||Key||Read size||Threads||preads/s off -> on||Speedup||p99 ms off -> 
on||Client CPU per read, us||
|EC rs-3-2|4 KB|8|14206 -> 24089|1.69x|2.07 -> 1.24|233 -> 118|
|EC rs-3-2|64 KB|1|2454 -> 3472|1.42x|0.82 -> 0.47|474 -> 244|
|EC rs-3-2|64 KB|8|11614 -> 18052|1.54x|2.46 -> 1.51|291 -> 163|
|EC rs-3-2|64 KB|32|20688 -> 27628|1.34x|5.19 -> 4.01|212 -> 139|
|EC rs-3-2|1 MB|8|3057 -> 3382|1.11x|5.50 -> 5.14|1196 -> 1018|
|RATIS/THREE|4 KB|8|14435 -> 23997|1.66x|2.11 -> 1.37|235 -> 116|
|RATIS/THREE|64 KB|8|11620 -> 18113|1.55x|1.94 -> 1.45|289 -> 162|
|RATIS/THREE|1 MB|8|3068 -> 3401|1.11x|5.86 -> 5.22|1197 -> 1019|

Profiles at 64 KB, 8 threads: client CPU per pread falls from 187 to 106 us, 
\{{initStreamRead}} and \{{closeStream}} drop by about 90%, datanode block-file 
opens drop 96%, and reader-thread \{{awaitReady}} wait falls from 18% to 0.9% 
of wall time. HBase reads 64 KB blocks with preads, so the 64 KB rows are the 
relevant ones.

h3. Compatibility / side effects

* None with the flag off (the default).
* No protocol or datanode change. Datanodes already accept several 
\{{ReadBlock}} requests per stream; the sequential reader relies on this.
* With the flag on, each idle reader keeps a datanode stream and an open block 
file on that datanode for up to the idle timeout, at most 2 per block stream. A 
client actively reading 1,000 EC rs-3-2 block groups can hold up to about 6,000 
extra open block files across the datanodes for up to 1 s.
* An idle reader keeps its \{{XceiverClientGrpc}} referenced, so 
\{{XceiverClientManager}} cannot close that client while the reader is idle. 
The idle timeout bounds this.

h3. Follow-ups (out of scope)

* *HDDS-17867:* deliver streaming pread replies on the waiting thread, removing 
the two thread hand-offs per response (about 15-20% of client CPU per pread 
with this reuse on).
* *Datanode re-opens the block file when a stream's block ID changes.* That 
would allow a per-channel pool shared across blocks, like HDDS-17561's, but it 
needs both sides and a version check.
* *Leftovers of the classic path, for HDDS-17561:* traced blocking calls still 
use one stream per call, because the datanode interceptor takes the span from 
call headers. Pooling is also skipped whenever any OpenTelemetry span is 
current. These are review items there, not part of this Jira.
* *HDDS-17716:* the per-call user lookup in \{{getPipelineCacheKey}} is still 
paid on every lease.

  was:
*Description*

*{*}Scope:{*}* the streaming read path (`StreamBlockInputStream`, stream read 
ON). The classic path's per-request stream cost is covered by
HDDS-17561 (pooled single-request streams for blocking ReadChunk/GetBlock), 
which does not touch `initStreamRead`/`streamRead`.

Relation to HDDS-17399 (lock-free positioned reads, PR #11724)
 - *{*}Supporting evidence.{*}* HDDS-17399's benchmark, after its lock removal, 
shows the stream path pread at 463 vs 1,488
ops/s for the chunk path (1 thread) and 1,975 vs 6,762 (8 threads), about 3.3× 
slower. The difference left is the
per-pread stream setup this Jira removes.
 - *{*}Constraints on the design:{*}*
1. The reader pool must be lock-free and must not take the stream monitor.
2. `closed` must be volatile and re-checked after a reader is returned. With 
17399, preads don't check `closed`, so
a pread finishing after `close()` must close its reader instead of pooling it.
3. An idle pooled reader must not hold an `XceiverClientGrpc` semaphore permit. 
There are 32 per client by default,
and the permit is held today from `initStreamRead` until `completeStreamRead`. 
Release it when the reader goes
idle and re-acquire it on lease.

Relation to HDDS-16396 / HDDS-16870
 - *{*}HDDS-16396{*}* (merged) made EC preads concurrent. The EC proxy builds a 
per-call clone that seeks, reads and closes.
 - *{*}HDDS-16870{*}* (branch `HDDS-16870`, not merged) works one layer 
*{*}up{*}*. On the healthy path, `ECBlockInputStream.readPositioned` reuses the
long-lived per-index block streams instead of cloning them, which removes the 
per-pread GetBlock (1.0–3.5 → ~0 per pread). It calls
`stream.read(position, slice)` → `BlockExtendedInputStream.readPositioned`.
 - *{*}When the index stream is a `StreamBlockInputStream`, HDDS-16870 does not 
change the per-call cost.{*}* That `readPositioned` still opens a fresh
one-shot gRPC stream per call (`initStreamRead` at 
`StreamBlockInputStream.java:216` on that branch). HDDS-16870 reuses the *Java 
stream
object*; this Jira reuses the {*}gRPC stream underneath it{*}.
 - The two are complementary. With both, an EC pread on a healthy key does no 
GetBlock and no new gRPC call.

Since HDDS-16396, `StreamBlockInputStream.readPositioned` serves every 
positioned read with a dedicated *{*}one-shot{*}* streaming call. That call
never touches the sequential reader or its pre-read buffer:
{code:java}
final XceiverClientGrpc client = acquireGrpcClientForReadData(factory, 
pipeline);
final StreamingReader reader = new StreamingReader(client);
client.initStreamRead(blockID, reader, failedStreamingDatanodes); // new gRPC 
call + semaphore permit
try {
return preadOnce(client, reader, pipeline, blockOffset, length, dst); // one 
ReadBlock request, poll until length
} finally {
closeStream(reader); // half-close, release permit
}
{code}
So every pread pays a full call lifecycle, which turns a streaming protocol 
into a unary one with extra steps:
 - `acquireClientForReadData` (with the per-call 
`UserGroupInformation.getCurrentUser()` lookup in
`XceiverClientManager.getPipelineCacheKey`, a separate Jira);
 - `initStreamRead`: `semaphore.acquire`, `stub.send`, the interceptor chain 
and HTTP/2 HEADERS;
 - `streamRead` waits for the new stream to become ready 
(`StreamingReadResponse.awaitReady`);
 - on the datanode, a new `ServerCall` and stream observer, plus 
`RandomAccessFileChannel.open` of the block file (`getChunkFile` stat + `open`);
 - `closeStream`: half-close, trailers, `completeStreamRead`; on the datanode, 
the block file is closed.

HBase issues most of its reads as preads (`BlockIOUtils.preadWithExtra`), 
including every read of a newly reopened store file.

Evidence (HBase YCSB on EC, stream read ON, RegionServer async-profiler JFR, 
6×60s)
 - *{*}Wall:{*}* streaming ReadBlock waits are 60.7% of busy RPC-handler time, 
and about half of that (48%) is preads: `preadOnce` → `StreamingReader.poll`.
`awaitReady` alone is ~4% of the streaming wait.
 - *{*}CPU, EC read path{*}* (`ECBlockInputStream.readFromStream|readCells` = 
100%): `readPositioned` is 66%. Of that:
 - `initStreamRead` (new call): 15%;
 - `closeStream`: 4.4%;
 - `streamRead` send: 16%, of which `awaitReady` is 5.5%;
 - `acquireGrpcClientForReadData` → `getPipelineCacheKey` user lookup: 18% 
(separate Jira).
 - *{*}Netty{*}* (47% of RS CPU in this arm): stream creation, HEADERS, and the 
`eventfd_write`/`unpark` wakeups per call.

*Proposed fix*

Keep a *{*}small pool of idle pread readers per `StreamBlockInputStream`{*}*, 
each bound to this block's stream on one datanode. Successive
positioned reads then send another `ReadBlock` on an already-open stream.

1. `readPositioned` takes an idle `StreamingReader` from the block's pool, or 
creates one with `initStreamRead` if none is idle. It runs
`preadOnce` and on success returns the reader to the pool. Concurrent preads 
(HDDS-16396) each hold their own reader, so a reader serves one
request at a time. That matches the datanode, which serves a stream 
sequentially (`requestLock`).
2. *{*}On any error, timeout, short read or EOF the reader is closed, never 
returned.{*}* Leftover responses for an abandoned request would otherwise
be read as data for the next one. `preadOnce` already validates each response's 
offset (`getByteBuffer(proto, blockOffset + copied)`); keep that
as a guard.
3. *{*}Bounded and time-limited:{*}*
 - at most `ozone.client.stream.read.pread.max-idle-readers` (default 2) idle 
readers per block stream;
 - idle readers are closed after `ozone.client.stream.read.pread.idle-timeout` 
(default ~1 s, below the datanode's
`streamReadFileIdleTimeout`);
 - all are closed on `unbuffer()`, `close()` and `refreshBlockInfo`.
An idle stream holds an `XceiverClientGrpc` semaphore permit, so the bound 
keeps permits available.
4. *{*}Failover is unchanged:{*}* a failed reader records its datanode in 
`failedStreamingDatanodes` before closing, and the next pread opens a new reader
on another datanode.
5. Behind `ozone.client.stream.read.pread.reuse.enabled` (default `false` 
first, then `true` after soak).

*Constraint: one block per stream*

A stream must stay bound to its block. The datanode opens the per-stream 
`blockFile` on the first `ReadBlock` and reuses it without checking later
requests' block IDs (`KeyValueHandler.readBlockImpl`: `if (!blockFile.isOpen()) 
blockFile.open(file)`). So readers cannot be shared across blocks
or across `StreamBlockInputStream`s. That is why the pool lives in the block 
stream, not in `XceiverClientGrpc`.

*Rejected / deferred alternatives*
 - *{*}Serve preads from the sequential `streamingReader`.{*}* Its queue holds 
pre-read data for the cursor position. Interleaving pread responses
would mean draining or discarding that window, which costs more than a new 
stream.
 - *{*}Multiplex concurrent preads on one stream.{*}* The datanode serves a 
stream sequentially, so concurrent preads would queue behind each other.
Responses also carry no request id, only offsets.
 - *{*}Fall back to unary ReadChunk for small preads.{*}* With HDDS-17561 a 
ReadChunk no longer opens a gRPC stream. But it needs the GetBlock chunk
list, which the streaming path otherwise avoids, and it would mix two read 
protocols on one block stream (retry, failover and
checksum-verification paths for both).
 - *{*}Datanode change to accept requests for any block on a stream{*}* (reopen 
`blockFile` when the block ID changes). This would enable a
per-datanode pool shared across blocks, like HDDS-17561's per-channel pool. 
It's a worthwhile follow-up, but it needs both sides and a
version check.

*Compatibility / side effects*
 - No proto or datanode change. Existing datanodes already accept several 
`ReadBlock` requests on one stream (the sequential reader does this).
 - Idle readers keep a datanode stream, a semaphore permit and an open block 
file on the datanode for up to the idle timeout.

*Tests*
 - Unit (`TestStreamBlockInputStream`, mocked `XceiverClientGrpc`):
 - N sequential preads on one block call `initStreamRead` once.
 - Concurrent preads use distinct readers.
 - Errors, timeouts and `OUT_OF_RANGE` close the reader, and the next pread 
opens a fresh one.
 - The pool size bound and idle timeout hold.
 - `unbuffer`/`close` release all permits.
 - Integration: EC and Ratis positioned reads with the flag on, including 
datanode failure mid-pread (failover to another datanode), token expiry
and refresh, and `readFully` interleaved with preads on the same stream.
 - Benchmark: 4 KB / 64 KB pread loop on an EC key (1/8/32 threads, TLS on), 
with client CPU per read, p50/p99, throughput and datanode CPU,
A/B n=3 per arm. Repeat the HBase YCSB scan profile and check that 
`initStreamRead`/`closeStream` drop out of the pread path.


> reuse streaming readers for positioned reads instead of one stream per 
> readPositioned
> -------------------------------------------------------------------------------------
>
>                 Key: HDDS-17752
>                 URL: https://issues.apache.org/jira/browse/HDDS-17752
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: EC Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> With streaming read on (\{{ozone.client.stream.readblock.enable=true}}), 
> \{{StreamBlockInputStream.readPositioned}} serves every positioned read with 
> its own one-shot streaming call. The call never touches the sequential reader 
> or its pre-read buffer:
> {code:java}
> final XceiverClientGrpc client = acquireGrpcClientForReadData(factory, 
> pipeline);
> final StreamingReader reader = new StreamingReader(client);
> client.initStreamRead(blockID, reader, failedStreamingDatanodes); // new gRPC 
> call + semaphore permit
> try {
> return preadOnce(client, reader, pipeline, blockOffset, length, dst); // one 
> ReadBlock request, poll until length
> } finally {
> closeStream(reader); // half-close, release permit
> }
> {code}
> So each pread pays a full call lifecycle, which turns a streaming protocol 
> into a unary one with extra steps:
> * a client acquire, including the per-call user lookup in 
> \{{XceiverClientManager.getPipelineCacheKey}} (HDDS-17716);
> * \{{initStreamRead}}: semaphore permit, a new HTTP/2 stream, HEADERS, the 
> interceptor chain;
> * \{{StreamingReadResponse.awaitReady}}, a wait until the new stream is 
> writable;
> * on the datanode, a new \{{ServerCall}} and stream observer, plus a 
> block-file open (\{{getChunkFile}} stat and open);
> * \{{closeStream}}: half-close, trailers, \{{completeStreamRead}}, and on the 
> datanode the block file is closed again.
> HBase issues most of its reads as preads (\{{BlockIOUtils.preadWithExtra}}), 
> including every read of a newly reopened store file, so this per-pread setup 
> dominates its streaming read path.
> h3. Evidence
> HBase YCSB on EC rs-3-2, streaming read on. RegionServer async-profiler, 6 x 
> 60 s:
> * *Wall:* streaming ReadBlock waits are 60.7% of busy RPC handler time, and 
> about half of that is preads (\{{preadOnce}} -> \{{StreamingReader.poll}}). 
> \{{awaitReady}} alone is about 4% of the streaming wait.
> * *CPU, EC read path* (\{{ECBlockInputStream}} read = 100%): 
> \{{readPositioned}} is 66%. Inside it, \{{initStreamRead}} is 15%, 
> \{{closeStream}} 4.4%, and the \{{streamRead}} send 16% (5.5% of it in 
> \{{awaitReady}}).
> * *Netty* threads are 47% of RegionServer CPU in this mode: stream creation, 
> HEADERS, and a wake-up per call.
> h3. What the related changes already cover
> * *HDDS-16396* (merged) made EC preads concurrent.
> * *HDDS-16870* (merged) makes healthy EC preads reuse the long-lived 
> per-index block streams instead of per-call clones. That removes the 
> per-pread GetBlock, and it means one \{{StreamBlockInputStream}} per EC index 
> now sees every pread on its block. It does not change what that stream does 
> per pread: \{{readPositioned}} still opens a one-shot gRPC call. HDDS-16870 
> reuses the Java stream object; this Jira reuses the gRPC stream underneath 
> it. With both, an EC pread on a healthy key does no GetBlock and opens no new 
> gRPC call.
> * *HDDS-17561* (Patch Available) pools single-request streams for blocking 
> \{{sendCommand}} calls (ReadChunk and GetBlock on the classic, non-streaming 
> read path) and delivers the reply on the waiting thread. It does not touch 
> \{{initStreamRead}} or \{{streamRead}}. Its per-channel pool also cannot hold 
> ReadBlock streams as they are, because a datanode binds a ReadBlock stream to 
> one block (see below). This Jira borrows its lifecycle rules (drop on error 
> or abandon, idle trim, client-side deadline) but needs its own per-block pool.
> * *HDDS-17399* (lock-free positioned reads) sets three constraints that this 
> design follows: the pool takes no stream monitor, \{{closed}} is volatile and 
> re-checked after a reader is pooled, and an idle reader holds no client 
> permit.
> h3. Proposed fix
> Keep a small pool of idle pread readers in each \{{StreamBlockInputStream}}. 
> After a pread that read everything it asked for, its stream stays open; the 
> next pread on the block takes the idle reader and sends one more 
> \{{ReadBlock}} on it.
> * *One reader per pread at a time.* Concurrent preads use different readers. 
> This matches the datanode, which serves a stream sequentially.
> * *Reuse only after a complete, clean read.* A reader goes back to the pool 
> only if it got the full length and its stream is still open with nothing 
> unread. The datanode's \{{BlockReadCursor}} sends contiguous ranges ending 
> with the one that covers the requested end, so a complete pread leaves 
> nothing in flight. Errors, short reads, EOF and \{{OUT_OF_RANGE}} close the 
> reader. \{{preadOnce}} still checks each response's offset as a guard.
> * *Idle readers hold no client permit.* The permit is returned when a reader 
> goes idle and taken again on lease (\{{XceiverClientGrpc.resumeStreamRead}}). 
> Otherwise idle readers would eat into the 32 permits per client.
> * *Lock-free pool:* a \{{ConcurrentLinkedDeque}} used most-recently-used 
> first, with a CAS-bounded count.
> * *Close, unbuffer and refresh close all idle readers.* A pool generation 
> advances on \{{unbuffer()}} and block-info refresh. A pread that started in 
> an older generation, or that finishes after \{{close()}}, closes its reader 
> instead of pooling it, so no reader of an old pipeline survives a refresh.
> * *A failed reused stream does not blame the datanode.* An idle stream may 
> have broken while pooled. Its first failure is retried once on a new stream 
> without adding the datanode to \{{failedStreamingDatanodes}}, and the other 
> idle readers are dropped. Timeouts and interrupts are not retried, so a hung 
> datanode still fails over after one read timeout.
> * *Bounded and time-limited*, behind a flag:
> ||Config||Default||Meaning||
> |ozone.client.stream.read.pread.reuse.enabled|false|turn the reuse on|
> |ozone.client.stream.read.pread.max-idle-readers|2|idle readers kept per 
> block stream|
> |ozone.client.stream.read.pread.idle-timeout|1s|close idle readers after 
> this, well below the datanode's 1 min stream file idle timeout|
> h3. Constraint: one block per stream
> The datanode opens the per-stream block file on the first \{{ReadBlock}} and 
> reuses it without checking the block ID of later requests 
> (\{{KeyValueHandler.readBlockImpl}} opens \{{blockFile}} only if it is not 
> open yet). So a reader cannot be shared across blocks or across 
> \{{StreamBlockInputStream}} instances, and the pool lives in the block 
> stream, not in \{{XceiverClientGrpc}}.
> h3. Rejected alternatives
> * *Serve preads from the sequential reader.* Its queue holds pre-read data 
> for the cursor position, so interleaved pread responses would mean draining 
> or discarding that window.
> * *Multiplex concurrent preads on one stream.* The datanode serves a stream 
> sequentially, and responses carry offsets but no request id.
> * *Unary ReadChunk for small preads.* With HDDS-17561 a ReadChunk no longer 
> opens a gRPC stream, but it needs the GetBlock chunk list, which the 
> streaming path otherwise avoids, and it mixes two read protocols on one block 
> stream.
> h3. Results
> A/B with only the flag toggled, same build. Secure compose cluster (Kerberos, 
> gRPC TLS, 5 datanodes), \{{ozone freon ocpr}} on 48 MB keys, 3 repetitions 
> per arm, 108 runs, 0 failures:
> ||Key||Read size||Threads||preads/s off -> on||Speedup||p99 ms off -> 
> on||Client CPU per read, us||
> |EC rs-3-2|4 KB|8|14206 -> 24089|1.69x|2.07 -> 1.24|233 -> 118|
> |EC rs-3-2|64 KB|1|2454 -> 3472|1.42x|0.82 -> 0.47|474 -> 244|
> |EC rs-3-2|64 KB|8|11614 -> 18052|1.54x|2.46 -> 1.51|291 -> 163|
> |EC rs-3-2|64 KB|32|20688 -> 27628|1.34x|5.19 -> 4.01|212 -> 139|
> |EC rs-3-2|1 MB|8|3057 -> 3382|1.11x|5.50 -> 5.14|1196 -> 1018|
> |RATIS/THREE|4 KB|8|14435 -> 23997|1.66x|2.11 -> 1.37|235 -> 116|
> |RATIS/THREE|64 KB|8|11620 -> 18113|1.55x|1.94 -> 1.45|289 -> 162|
> |RATIS/THREE|1 MB|8|3068 -> 3401|1.11x|5.86 -> 5.22|1197 -> 1019|
> Profiles at 64 KB, 8 threads: client CPU per pread falls from 187 to 106 us, 
> \{{initStreamRead}} and \{{closeStream}} drop by about 90%, datanode 
> block-file opens drop 96%, and reader-thread \{{awaitReady}} wait falls from 
> 18% to 0.9% of wall time. HBase reads 64 KB blocks with preads, so the 64 KB 
> rows are the relevant ones.
> h3. Compatibility / side effects
> * None with the flag off (the default).
> * No protocol or datanode change. Datanodes already accept several 
> \{{ReadBlock}} requests per stream; the sequential reader relies on this.
> * With the flag on, each idle reader keeps a datanode stream and an open 
> block file on that datanode for up to the idle timeout, at most 2 per block 
> stream. A client actively reading 1,000 EC rs-3-2 block groups can hold up to 
> about 6,000 extra open block files across the datanodes for up to 1 s.
> * An idle reader keeps its \{{XceiverClientGrpc}} referenced, so 
> \{{XceiverClientManager}} cannot close that client while the reader is idle. 
> The idle timeout bounds this.
> h3. Follow-ups (out of scope)
> * *HDDS-17867:* deliver streaming pread replies on the waiting thread, 
> removing the two thread hand-offs per response (about 15-20% of client CPU 
> per pread with this reuse on).
> * *Datanode re-opens the block file when a stream's block ID changes.* That 
> would allow a per-channel pool shared across blocks, like HDDS-17561's, but 
> it needs both sides and a version check.
> * *Leftovers of the classic path, for HDDS-17561:* traced blocking calls 
> still use one stream per call, because the datanode interceptor takes the 
> span from call headers. Pooling is also skipped whenever any OpenTelemetry 
> span is current. These are review items there, not part of this Jira.
> * *HDDS-17716:* the per-call user lookup in \{{getPipelineCacheKey}} is still 
> paid on every lease.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to