Andrey Yarovoy created HDDS-17867:
-------------------------------------

             Summary: Remove the two thread hand-offs per response in streaming 
positioned reads
                 Key: HDDS-17867
                 URL: https://issues.apache.org/jira/browse/HDDS-17867
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Client
            Reporter: Andrey Yarovoy


After HDDS-17752 a positioned streaming read no longer opens a stream per call. 
Profiles of the remaining per-read cost show two thread hand-offs for every 
`ReadBlock` response.

1. **gRPC callback executor.** `XceiverClientGrpc.initStreamRead` starts the 
call with
`stub.send(new ReadyAwareResponseObserver(...))` without `withExecutor`. So 
every callback (`onMessage`, `onReady`,
`onClose`) is passed from the netty event loop to gRPC's shared default 
executor, a cached pool with a
`SynchronousQueue`. That pool thread deframes and parses the response and runs 
`StreamingReader.onNext`, which
verifies the checksums.
2. **Reader queue.** `StreamingReader.onNext` then puts the response on a 
`LinkedBlockingQueue` (`offerToQueue`). That
wakes the pread thread blocked in `StreamingReader.poll()`, which copies the 
data out.

So each response costs two wake-ups (`unpark` → `pthread_cond_signal` → futex) 
and runs on a third thread, while the
pread thread that needs the data is idle in `poll()`.

### Evidence

Secure compose cluster (Kerberos, gRPC TLS, 5 datanodes), freon `ocpr`, EC 
rs-3-2-1024k, 64 KB preads, 8 threads,
HDDS-17752 reuse on. async-profiler `cpu`, client side, µs of CPU per pread 
(client total ≈ 106 µs):

| cost | µs/pread |
|---|---|
| `pthread_cond_signal` (wake-ups): `SynchronousQueue.xferLifo` 5.9 + 
`AQS.signalNext` (reader queue) 5.3 | 11.2 |
| futex waits behind those hand-offs | 3.8 |
| executor threads: deframe + protobuf parse | 6.0 |
| executor threads: `offerToQueue` (enqueue + wake the reader) | 5.7 |
| executor threads: idle / hand-off (`park`, `SynchronousQueue` poll) | 3.2 |

The wake-ups alone are about 10% of client CPU per read. On a loaded 
RegionServer each hand-off also adds scheduling
latency to every read.

## Proposed fix

For **positioned** reads only, make the pread thread the call's executor, as 
HDDS-17561 does for blocking unary calls:

- The pread `StreamingReader`, pooled since HDDS-17752, implements `Executor`. 
While a pread is waiting, callbacks are
queued to it and run by the pread thread inside `poll()`. That covers 
deframing, protobuf parsing, checksum
verification and the data copy.
- When no pread is waiting (the reader is idle in the pool), callbacks run 
directly on the calling thread. These are
only `onClose`/`onReady`, with no data.
- The response queue then disappears: `onNext` hands the response to the 
waiting `poll()` on the same thread.
- The **sequential** reader keeps today's executor and queue. Its pre-read 
window relies on responses being consumed
while the application processes earlier data. Running callbacks only when the 
reader polls would hold back gRPC
inbound flow control and shrink the effective pre-read.

Same rules as HDDS-17561's `CallerRunsReplyFuture`:
- a waiter that times out, or is interrupted, cancels the stream;
- callbacks queued after the waiter left run directly;
- `close()` must not leave callbacks queued on a reader nobody will poll.

## Compatibility / side effects

- No protocol or datanode change. The checksum verification and protobuf 
parsing move from a pool thread to the pread
thread, so the pread thread does that CPU work itself instead of waiting.
- Concurrent preads on one block already use separate readers (HDDS-17752), so 
there is no cross-request
head-of-line blocking.

## Tests

- **Unit** (`TestStreamBlockInputStream`):
- with a pooled reader, a response is processed on the pread thread;
- no callback is left queued after a timeout, an interrupt or `close()`;
- the sequential path is unchanged.
- **Integration:** EC and RATIS positioned reads with failover, with stream 
reuse on.
- **Benchmark:**
- the HDDS-17752 harnesses: `TestStreamPositionedReadBenchmark`, plus freon 
`ocpr` on the secure compose cluster;
- async-profiler: `SynchronousQueue.xferLifo` and 
`StreamingReader.offerToQueue` should drop out of the pread path.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to