[ 
https://issues.apache.org/jira/browse/HDDS-17752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17752:
----------------------------------
    Description: 
*Description*

*{*}Scope:{*}* the streaming read path (`StreamBlockInputStream`, stream read 
ON). The classic path's per-request stream cost is covered by
HDDS-17561 (pooled single-request streams for blocking ReadChunk/GetBlock), 
which does not touch `initStreamRead`/`streamRead`.

Relation to HDDS-17399 (lock-free positioned reads, PR #11724)
 - *{*}Supporting evidence.{*}* HDDS-17399's benchmark, after its lock removal, 
shows the stream path pread at 463 vs 1,488
ops/s for the chunk path (1 thread) and 1,975 vs 6,762 (8 threads), about 3.3× 
slower. The difference left is the
per-pread stream setup this Jira removes.
 - *{*}Constraints on the design:{*}*
1. The reader pool must be lock-free and must not take the stream monitor.
2. `closed` must be volatile and re-checked after a reader is returned. With 
17399, preads don't check `closed`, so
a pread finishing after `close()` must close its reader instead of pooling it.
3. An idle pooled reader must not hold an `XceiverClientGrpc` semaphore permit. 
There are 32 per client by default,
and the permit is held today from `initStreamRead` until `completeStreamRead`. 
Release it when the reader goes
idle and re-acquire it on lease.

Relation to HDDS-16396 / HDDS-16870
 - *{*}HDDS-16396{*}* (merged) made EC preads concurrent. The EC proxy builds a 
per-call clone that seeks, reads and closes.
 - *{*}HDDS-16870{*}* (branch `HDDS-16870`, not merged) works one layer 
*{*}up{*}*. On the healthy path, `ECBlockInputStream.readPositioned` reuses the
long-lived per-index block streams instead of cloning them, which removes the 
per-pread GetBlock (1.0–3.5 → ~0 per pread). It calls
`stream.read(position, slice)` → `BlockExtendedInputStream.readPositioned`.
 - *{*}When the index stream is a `StreamBlockInputStream`, HDDS-16870 does not 
change the per-call cost.{*}* That `readPositioned` still opens a fresh
one-shot gRPC stream per call (`initStreamRead` at 
`StreamBlockInputStream.java:216` on that branch). HDDS-16870 reuses the *Java 
stream
object*; this Jira reuses the {*}gRPC stream underneath it{*}.
 - The two are complementary. With both, an EC pread on a healthy key does no 
GetBlock and no new gRPC call.

Since HDDS-16396, `StreamBlockInputStream.readPositioned` serves every 
positioned read with a dedicated *{*}one-shot{*}* streaming call. That call
never touches the sequential reader or its pre-read buffer:
{code:java}
final XceiverClientGrpc client = acquireGrpcClientForReadData(factory, 
pipeline);
final StreamingReader reader = new StreamingReader(client);
client.initStreamRead(blockID, reader, failedStreamingDatanodes); // new gRPC 
call + semaphore permit
try {
return preadOnce(client, reader, pipeline, blockOffset, length, dst); // one 
ReadBlock request, poll until length
} finally {
closeStream(reader); // half-close, release permit
}
{code}
So every pread pays a full call lifecycle, which turns a streaming protocol 
into a unary one with extra steps:
 - `acquireClientForReadData` (with the per-call 
`UserGroupInformation.getCurrentUser()` lookup in
`XceiverClientManager.getPipelineCacheKey`, a separate Jira);
 - `initStreamRead`: `semaphore.acquire`, `stub.send`, the interceptor chain 
and HTTP/2 HEADERS;
 - `streamRead` waits for the new stream to become ready 
(`StreamingReadResponse.awaitReady`);
 - on the datanode, a new `ServerCall` and stream observer, plus 
`RandomAccessFileChannel.open` of the block file (`getChunkFile` stat + `open`);
 - `closeStream`: half-close, trailers, `completeStreamRead`; on the datanode, 
the block file is closed.

HBase issues most of its reads as preads (`BlockIOUtils.preadWithExtra`), 
including every read of a newly reopened store file.

Evidence (HBase YCSB on EC, stream read ON, RegionServer async-profiler JFR, 
6×60s)
 - *{*}Wall:{*}* streaming ReadBlock waits are 60.7% of busy RPC-handler time, 
and about half of that (48%) is preads: `preadOnce` → `StreamingReader.poll`.
`awaitReady` alone is ~4% of the streaming wait.
 - *{*}CPU, EC read path{*}* (`ECBlockInputStream.readFromStream|readCells` = 
100%): `readPositioned` is 66%. Of that:
 - `initStreamRead` (new call): 15%;
 - `closeStream`: 4.4%;
 - `streamRead` send: 16%, of which `awaitReady` is 5.5%;
 - `acquireGrpcClientForReadData` → `getPipelineCacheKey` user lookup: 18% 
(separate Jira).
 - *{*}Netty{*}* (47% of RS CPU in this arm): stream creation, HEADERS, and the 
`eventfd_write`/`unpark` wakeups per call.

*Proposed fix*

Keep a *{*}small pool of idle pread readers per `StreamBlockInputStream`{*}*, 
each bound to this block's stream on one datanode. Successive
positioned reads then send another `ReadBlock` on an already-open stream.

1. `readPositioned` takes an idle `StreamingReader` from the block's pool, or 
creates one with `initStreamRead` if none is idle. It runs
`preadOnce` and on success returns the reader to the pool. Concurrent preads 
(HDDS-16396) each hold their own reader, so a reader serves one
request at a time. That matches the datanode, which serves a stream 
sequentially (`requestLock`).
2. *{*}On any error, timeout, short read or EOF the reader is closed, never 
returned.{*}* Leftover responses for an abandoned request would otherwise
be read as data for the next one. `preadOnce` already validates each response's 
offset (`getByteBuffer(proto, blockOffset + copied)`); keep that
as a guard.
3. *{*}Bounded and time-limited:{*}*
 - at most `ozone.client.stream.read.pread.max-idle-readers` (default 2) idle 
readers per block stream;
 - idle readers are closed after `ozone.client.stream.read.pread.idle-timeout` 
(default ~1 s, below the datanode's
`streamReadFileIdleTimeout`);
 - all are closed on `unbuffer()`, `close()` and `refreshBlockInfo`.
An idle stream holds an `XceiverClientGrpc` semaphore permit, so the bound 
keeps permits available.
4. *{*}Failover is unchanged:{*}* a failed reader records its datanode in 
`failedStreamingDatanodes` before closing, and the next pread opens a new reader
on another datanode.
5. Behind `ozone.client.stream.read.pread.reuse.enabled` (default `false` 
first, then `true` after soak).

*Constraint: one block per stream*

A stream must stay bound to its block. The datanode opens the per-stream 
`blockFile` on the first `ReadBlock` and reuses it without checking later
requests' block IDs (`KeyValueHandler.readBlockImpl`: `if (!blockFile.isOpen()) 
blockFile.open(file)`). So readers cannot be shared across blocks
or across `StreamBlockInputStream`s. That is why the pool lives in the block 
stream, not in `XceiverClientGrpc`.

*Rejected / deferred alternatives*
 - *{*}Serve preads from the sequential `streamingReader`.{*}* Its queue holds 
pre-read data for the cursor position. Interleaving pread responses
would mean draining or discarding that window, which costs more than a new 
stream.
 - *{*}Multiplex concurrent preads on one stream.{*}* The datanode serves a 
stream sequentially, so concurrent preads would queue behind each other.
Responses also carry no request id, only offsets.
 - *{*}Fall back to unary ReadChunk for small preads.{*}* With HDDS-17561 a 
ReadChunk no longer opens a gRPC stream. But it needs the GetBlock chunk
list, which the streaming path otherwise avoids, and it would mix two read 
protocols on one block stream (retry, failover and
checksum-verification paths for both).
 - *{*}Datanode change to accept requests for any block on a stream{*}* (reopen 
`blockFile` when the block ID changes). This would enable a
per-datanode pool shared across blocks, like HDDS-17561's per-channel pool. 
It's a worthwhile follow-up, but it needs both sides and a
version check.

*Compatibility / side effects*
 - No proto or datanode change. Existing datanodes already accept several 
`ReadBlock` requests on one stream (the sequential reader does this).
 - Idle readers keep a datanode stream, a semaphore permit and an open block 
file on the datanode for up to the idle timeout.

*Tests*
 - Unit (`TestStreamBlockInputStream`, mocked `XceiverClientGrpc`):
 - N sequential preads on one block call `initStreamRead` once.
 - Concurrent preads use distinct readers.
 - Errors, timeouts and `OUT_OF_RANGE` close the reader, and the next pread 
opens a fresh one.
 - The pool size bound and idle timeout hold.
 - `unbuffer`/`close` release all permits.
 - Integration: EC and Ratis positioned reads with the flag on, including 
datanode failure mid-pread (failover to another datanode), token expiry
and refresh, and `readFully` interleaved with preads on the same stream.
 - Benchmark: 4 KB / 64 KB pread loop on an EC key (1/8/32 threads, TLS on), 
with client CPU per read, p50/p99, throughput and datanode CPU,
A/B n=3 per arm. Repeat the HBase YCSB scan profile and check that 
`initStreamRead`/`closeStream` drop out of the pread path.

  was:
## Description

**Scope:** the streaming read path (`StreamBlockInputStream`, stream read ON). 
The classic path's per-request stream cost is covered by
HDDS-17561 (pooled single-request streams for blocking ReadChunk/GetBlock), 
which does not touch `initStreamRead`/`streamRead`.

### Relation to HDDS-17399 (lock-free positioned reads, PR #11724)

- **Supporting evidence.** HDDS-17399's benchmark, after its lock removal, 
shows the stream path pread at 463 vs 1,488
ops/s for the chunk path (1 thread) and 1,975 vs 6,762 (8 threads), about 3.3× 
slower. The difference left is the
per-pread stream setup this Jira removes.
- **Constraints on the design:**
1. The reader pool must be lock-free and must not take the stream monitor.
2. `closed` must be volatile and re-checked after a reader is returned. With 
17399, preads don't check `closed`, so
a pread finishing after `close()` must close its reader instead of pooling it.
3. An idle pooled reader must not hold an `XceiverClientGrpc` semaphore permit. 
There are 32 per client by default,
and the permit is held today from `initStreamRead` until `completeStreamRead`. 
Release it when the reader goes
idle and re-acquire it on lease.

See `interplay-17399.md`.

### Relation to HDDS-16396 / HDDS-16870

- **HDDS-16396** (merged) made EC preads concurrent. The EC proxy builds a 
per-call clone that seeks, reads and closes.
- **HDDS-16870** (branch `HDDS-16870`, not merged) works one layer **up**. On 
the healthy path, `ECBlockInputStream.readPositioned` reuses the
long-lived per-index block streams instead of cloning them, which removes the 
per-pread GetBlock (1.0–3.5 → ~0 per pread). It calls
`stream.read(position, slice)` → `BlockExtendedInputStream.readPositioned`.
- **When the index stream is a `StreamBlockInputStream`, HDDS-16870 does not 
change the per-call cost.** That `readPositioned` still opens a fresh
one-shot gRPC stream per call (`initStreamRead` at 
`StreamBlockInputStream.java:216` on that branch). HDDS-16870 reuses the *Java 
stream
object*; this Jira reuses the *gRPC stream underneath it*.
- The two are complementary. With both, an EC pread on a healthy key does no 
GetBlock and no new gRPC call.

Since HDDS-16396, `StreamBlockInputStream.readPositioned` serves every 
positioned read with a dedicated **one-shot** streaming call. That call
never touches the sequential reader or its pre-read buffer:

{code:java}
final XceiverClientGrpc client = acquireGrpcClientForReadData(factory, 
pipeline);
final StreamingReader reader = new StreamingReader(client);
client.initStreamRead(blockID, reader, failedStreamingDatanodes); // new gRPC 
call + semaphore permit
try {
return preadOnce(client, reader, pipeline, blockOffset, length, dst); // one 
ReadBlock request, poll until length
} finally {
closeStream(reader); // half-close, release permit
}
{code}

So every pread pays a full call lifecycle, which turns a streaming protocol 
into a unary one with extra steps:
- `acquireClientForReadData` (with the per-call 
`UserGroupInformation.getCurrentUser()` lookup in
`XceiverClientManager.getPipelineCacheKey`, a separate Jira);
- `initStreamRead`: `semaphore.acquire`, `stub.send`, the interceptor chain and 
HTTP/2 HEADERS;
- `streamRead` waits for the new stream to become ready 
(`StreamingReadResponse.awaitReady`);
- on the datanode, a new `ServerCall` and stream observer, plus 
`RandomAccessFileChannel.open` of the block file (`getChunkFile` stat + `open`);
- `closeStream`: half-close, trailers, `completeStreamRead`; on the datanode, 
the block file is closed.

HBase issues most of its reads as preads (`BlockIOUtils.preadWithExtra`), 
including every read of a newly reopened store file.

### Evidence (HBase YCSB on EC, stream read ON, RegionServer async-profiler 
JFR, 6×60s)

- **Wall:** streaming ReadBlock waits are 60.7% of busy RPC-handler time, and 
about half of that (48%) is preads: `preadOnce` → `StreamingReader.poll`.
`awaitReady` alone is ~4% of the streaming wait.
- **CPU, EC read path** (`ECBlockInputStream.readFromStream|readCells` = 100%): 
`readPositioned` is 66%. Of that:
- `initStreamRead` (new call): 15%;
- `closeStream`: 4.4%;
- `streamRead` send: 16%, of which `awaitReady` is 5.5%;
- `acquireGrpcClientForReadData` → `getPipelineCacheKey` user lookup: 18% 
(separate Jira).
- **Netty** (47% of RS CPU in this arm): stream creation, HEADERS, and the 
`eventfd_write`/`unpark` wakeups per call.

## Proposed fix

Keep a **small pool of idle pread readers per `StreamBlockInputStream`**, each 
bound to this block's stream on one datanode. Successive
positioned reads then send another `ReadBlock` on an already-open stream.

1. `readPositioned` takes an idle `StreamingReader` from the block's pool, or 
creates one with `initStreamRead` if none is idle. It runs
`preadOnce` and on success returns the reader to the pool. Concurrent preads 
(HDDS-16396) each hold their own reader, so a reader serves one
request at a time. That matches the datanode, which serves a stream 
sequentially (`requestLock`).
2. **On any error, timeout, short read or EOF the reader is closed, never 
returned.** Leftover responses for an abandoned request would otherwise
be read as data for the next one. `preadOnce` already validates each response's 
offset (`getByteBuffer(proto, blockOffset + copied)`); keep that
as a guard.
3. **Bounded and time-limited:**
- at most `ozone.client.stream.read.pread.max-idle-readers` (default 2) idle 
readers per block stream;
- idle readers are closed after `ozone.client.stream.read.pread.idle-timeout` 
(default ~1 s, below the datanode's
`streamReadFileIdleTimeout`);
- all are closed on `unbuffer()`, `close()` and `refreshBlockInfo`.
An idle stream holds an `XceiverClientGrpc` semaphore permit, so the bound 
keeps permits available.
4. **Failover is unchanged:** a failed reader records its datanode in 
`failedStreamingDatanodes` before closing, and the next pread opens a new reader
on another datanode.
5. Behind `ozone.client.stream.read.pread.reuse.enabled` (default `false` 
first, then `true` after soak).

### Constraint: one block per stream

A stream must stay bound to its block. The datanode opens the per-stream 
`blockFile` on the first `ReadBlock` and reuses it without checking later
requests' block IDs (`KeyValueHandler.readBlockImpl`: `if (!blockFile.isOpen()) 
blockFile.open(file)`). So readers cannot be shared across blocks
or across `StreamBlockInputStream`s. That is why the pool lives in the block 
stream, not in `XceiverClientGrpc`.

### Rejected / deferred alternatives

- **Serve preads from the sequential `streamingReader`.** Its queue holds 
pre-read data for the cursor position. Interleaving pread responses
would mean draining or discarding that window, which costs more than a new 
stream.
- **Multiplex concurrent preads on one stream.** The datanode serves a stream 
sequentially, so concurrent preads would queue behind each other.
Responses also carry no request id, only offsets.
- **Fall back to unary ReadChunk for small preads.** With HDDS-17561 a 
ReadChunk no longer opens a gRPC stream. But it needs the GetBlock chunk
list, which the streaming path otherwise avoids, and it would mix two read 
protocols on one block stream (retry, failover and
checksum-verification paths for both).
- **Datanode change to accept requests for any block on a stream** (reopen 
`blockFile` when the block ID changes). This would enable a
per-datanode pool shared across blocks, like HDDS-17561's per-channel pool. 
It's a worthwhile follow-up, but it needs both sides and a
version check.

## Compatibility / side effects

- No proto or datanode change. Existing datanodes already accept several 
`ReadBlock` requests on one stream (the sequential reader does this).
- Idle readers keep a datanode stream, a semaphore permit and an open block 
file on the datanode for up to the idle timeout.

## Tests

- Unit (`TestStreamBlockInputStream`, mocked `XceiverClientGrpc`):
- N sequential preads on one block call `initStreamRead` once.
- Concurrent preads use distinct readers.
- Errors, timeouts and `OUT_OF_RANGE` close the reader, and the next pread 
opens a fresh one.
- The pool size bound and idle timeout hold.
- `unbuffer`/`close` release all permits.
- Integration: EC and Ratis positioned reads with the flag on, including 
datanode failure mid-pread (failover to another datanode), token expiry
and refresh, and `readFully` interleaved with preads on the same stream.
- Benchmark: 4 KB / 64 KB pread loop on an EC key (1/8/32 threads, TLS on), 
with client CPU per read, p50/p99, throughput and datanode CPU,
A/B n=3 per arm. Repeat the HBase YCSB scan profile and check that 
`initStreamRead`/`closeStream` drop out of the pread path.


> reuse streaming readers for positioned reads instead of one stream per 
> readPositioned
> -------------------------------------------------------------------------------------
>
>                 Key: HDDS-17752
>                 URL: https://issues.apache.org/jira/browse/HDDS-17752
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: EC Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> *Description*
> *{*}Scope:{*}* the streaming read path (`StreamBlockInputStream`, stream read 
> ON). The classic path's per-request stream cost is covered by
> HDDS-17561 (pooled single-request streams for blocking ReadChunk/GetBlock), 
> which does not touch `initStreamRead`/`streamRead`.
> Relation to HDDS-17399 (lock-free positioned reads, PR #11724)
>  - *{*}Supporting evidence.{*}* HDDS-17399's benchmark, after its lock 
> removal, shows the stream path pread at 463 vs 1,488
> ops/s for the chunk path (1 thread) and 1,975 vs 6,762 (8 threads), about 
> 3.3× slower. The difference left is the
> per-pread stream setup this Jira removes.
>  - *{*}Constraints on the design:{*}*
> 1. The reader pool must be lock-free and must not take the stream monitor.
> 2. `closed` must be volatile and re-checked after a reader is returned. With 
> 17399, preads don't check `closed`, so
> a pread finishing after `close()` must close its reader instead of pooling it.
> 3. An idle pooled reader must not hold an `XceiverClientGrpc` semaphore 
> permit. There are 32 per client by default,
> and the permit is held today from `initStreamRead` until 
> `completeStreamRead`. Release it when the reader goes
> idle and re-acquire it on lease.
> Relation to HDDS-16396 / HDDS-16870
>  - *{*}HDDS-16396{*}* (merged) made EC preads concurrent. The EC proxy builds 
> a per-call clone that seeks, reads and closes.
>  - *{*}HDDS-16870{*}* (branch `HDDS-16870`, not merged) works one layer 
> *{*}up{*}*. On the healthy path, `ECBlockInputStream.readPositioned` reuses 
> the
> long-lived per-index block streams instead of cloning them, which removes the 
> per-pread GetBlock (1.0–3.5 → ~0 per pread). It calls
> `stream.read(position, slice)` → `BlockExtendedInputStream.readPositioned`.
>  - *{*}When the index stream is a `StreamBlockInputStream`, HDDS-16870 does 
> not change the per-call cost.{*}* That `readPositioned` still opens a fresh
> one-shot gRPC stream per call (`initStreamRead` at 
> `StreamBlockInputStream.java:216` on that branch). HDDS-16870 reuses the 
> *Java stream
> object*; this Jira reuses the {*}gRPC stream underneath it{*}.
>  - The two are complementary. With both, an EC pread on a healthy key does no 
> GetBlock and no new gRPC call.
> Since HDDS-16396, `StreamBlockInputStream.readPositioned` serves every 
> positioned read with a dedicated *{*}one-shot{*}* streaming call. That call
> never touches the sequential reader or its pre-read buffer:
> {code:java}
> final XceiverClientGrpc client = acquireGrpcClientForReadData(factory, 
> pipeline);
> final StreamingReader reader = new StreamingReader(client);
> client.initStreamRead(blockID, reader, failedStreamingDatanodes); // new gRPC 
> call + semaphore permit
> try {
> return preadOnce(client, reader, pipeline, blockOffset, length, dst); // one 
> ReadBlock request, poll until length
> } finally {
> closeStream(reader); // half-close, release permit
> }
> {code}
> So every pread pays a full call lifecycle, which turns a streaming protocol 
> into a unary one with extra steps:
>  - `acquireClientForReadData` (with the per-call 
> `UserGroupInformation.getCurrentUser()` lookup in
> `XceiverClientManager.getPipelineCacheKey`, a separate Jira);
>  - `initStreamRead`: `semaphore.acquire`, `stub.send`, the interceptor chain 
> and HTTP/2 HEADERS;
>  - `streamRead` waits for the new stream to become ready 
> (`StreamingReadResponse.awaitReady`);
>  - on the datanode, a new `ServerCall` and stream observer, plus 
> `RandomAccessFileChannel.open` of the block file (`getChunkFile` stat + 
> `open`);
>  - `closeStream`: half-close, trailers, `completeStreamRead`; on the 
> datanode, the block file is closed.
> HBase issues most of its reads as preads (`BlockIOUtils.preadWithExtra`), 
> including every read of a newly reopened store file.
> Evidence (HBase YCSB on EC, stream read ON, RegionServer async-profiler JFR, 
> 6×60s)
>  - *{*}Wall:{*}* streaming ReadBlock waits are 60.7% of busy RPC-handler 
> time, and about half of that (48%) is preads: `preadOnce` → 
> `StreamingReader.poll`.
> `awaitReady` alone is ~4% of the streaming wait.
>  - *{*}CPU, EC read path{*}* (`ECBlockInputStream.readFromStream|readCells` = 
> 100%): `readPositioned` is 66%. Of that:
>  - `initStreamRead` (new call): 15%;
>  - `closeStream`: 4.4%;
>  - `streamRead` send: 16%, of which `awaitReady` is 5.5%;
>  - `acquireGrpcClientForReadData` → `getPipelineCacheKey` user lookup: 18% 
> (separate Jira).
>  - *{*}Netty{*}* (47% of RS CPU in this arm): stream creation, HEADERS, and 
> the `eventfd_write`/`unpark` wakeups per call.
> *Proposed fix*
> Keep a *{*}small pool of idle pread readers per `StreamBlockInputStream`{*}*, 
> each bound to this block's stream on one datanode. Successive
> positioned reads then send another `ReadBlock` on an already-open stream.
> 1. `readPositioned` takes an idle `StreamingReader` from the block's pool, or 
> creates one with `initStreamRead` if none is idle. It runs
> `preadOnce` and on success returns the reader to the pool. Concurrent preads 
> (HDDS-16396) each hold their own reader, so a reader serves one
> request at a time. That matches the datanode, which serves a stream 
> sequentially (`requestLock`).
> 2. *{*}On any error, timeout, short read or EOF the reader is closed, never 
> returned.{*}* Leftover responses for an abandoned request would otherwise
> be read as data for the next one. `preadOnce` already validates each 
> response's offset (`getByteBuffer(proto, blockOffset + copied)`); keep that
> as a guard.
> 3. *{*}Bounded and time-limited:{*}*
>  - at most `ozone.client.stream.read.pread.max-idle-readers` (default 2) idle 
> readers per block stream;
>  - idle readers are closed after 
> `ozone.client.stream.read.pread.idle-timeout` (default ~1 s, below the 
> datanode's
> `streamReadFileIdleTimeout`);
>  - all are closed on `unbuffer()`, `close()` and `refreshBlockInfo`.
> An idle stream holds an `XceiverClientGrpc` semaphore permit, so the bound 
> keeps permits available.
> 4. *{*}Failover is unchanged:{*}* a failed reader records its datanode in 
> `failedStreamingDatanodes` before closing, and the next pread opens a new 
> reader
> on another datanode.
> 5. Behind `ozone.client.stream.read.pread.reuse.enabled` (default `false` 
> first, then `true` after soak).
> *Constraint: one block per stream*
> A stream must stay bound to its block. The datanode opens the per-stream 
> `blockFile` on the first `ReadBlock` and reuses it without checking later
> requests' block IDs (`KeyValueHandler.readBlockImpl`: `if 
> (!blockFile.isOpen()) blockFile.open(file)`). So readers cannot be shared 
> across blocks
> or across `StreamBlockInputStream`s. That is why the pool lives in the block 
> stream, not in `XceiverClientGrpc`.
> *Rejected / deferred alternatives*
>  - *{*}Serve preads from the sequential `streamingReader`.{*}* Its queue 
> holds pre-read data for the cursor position. Interleaving pread responses
> would mean draining or discarding that window, which costs more than a new 
> stream.
>  - *{*}Multiplex concurrent preads on one stream.{*}* The datanode serves a 
> stream sequentially, so concurrent preads would queue behind each other.
> Responses also carry no request id, only offsets.
>  - *{*}Fall back to unary ReadChunk for small preads.{*}* With HDDS-17561 a 
> ReadChunk no longer opens a gRPC stream. But it needs the GetBlock chunk
> list, which the streaming path otherwise avoids, and it would mix two read 
> protocols on one block stream (retry, failover and
> checksum-verification paths for both).
>  - *{*}Datanode change to accept requests for any block on a stream{*}* 
> (reopen `blockFile` when the block ID changes). This would enable a
> per-datanode pool shared across blocks, like HDDS-17561's per-channel pool. 
> It's a worthwhile follow-up, but it needs both sides and a
> version check.
> *Compatibility / side effects*
>  - No proto or datanode change. Existing datanodes already accept several 
> `ReadBlock` requests on one stream (the sequential reader does this).
>  - Idle readers keep a datanode stream, a semaphore permit and an open block 
> file on the datanode for up to the idle timeout.
> *Tests*
>  - Unit (`TestStreamBlockInputStream`, mocked `XceiverClientGrpc`):
>  - N sequential preads on one block call `initStreamRead` once.
>  - Concurrent preads use distinct readers.
>  - Errors, timeouts and `OUT_OF_RANGE` close the reader, and the next pread 
> opens a fresh one.
>  - The pool size bound and idle timeout hold.
>  - `unbuffer`/`close` release all permits.
>  - Integration: EC and Ratis positioned reads with the flag on, including 
> datanode failure mid-pread (failover to another datanode), token expiry
> and refresh, and `readFully` interleaved with preads on the same stream.
>  - Benchmark: 4 KB / 64 KB pread loop on an EC key (1/8/32 threads, TLS on), 
> with client CPU per read, p50/p99, throughput and datanode CPU,
> A/B n=3 per arm. Repeat the HBase YCSB scan profile and check that 
> `initStreamRead`/`closeStream` drop out of the pread path.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to