[ 
https://issues.apache.org/jira/browse/HDDS-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17867:
----------------------------------
    Labels: ozone-hbase-perf  (was: )

> Remove the two thread hand-offs per response in streaming positioned reads
> --------------------------------------------------------------------------
>
>                 Key: HDDS-17867
>                 URL: https://issues.apache.org/jira/browse/HDDS-17867
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>              Labels: ozone-hbase-perf
>
> After HDDS-17752 a positioned streaming read no longer opens a stream per 
> call. Profiles of the remaining per-read cost show two thread hand-offs for 
> every `ReadBlock` response.
> 1. **gRPC callback executor.** `XceiverClientGrpc.initStreamRead` starts the 
> call with
> `stub.send(new ReadyAwareResponseObserver(...))` without `withExecutor`. So 
> every callback (`onMessage`, `onReady`,
> `onClose`) is passed from the netty event loop to gRPC's shared default 
> executor, a cached pool with a
> `SynchronousQueue`. That pool thread deframes and parses the response and 
> runs `StreamingReader.onNext`, which
> verifies the checksums.
> 2. **Reader queue.** `StreamingReader.onNext` then puts the response on a 
> `LinkedBlockingQueue` (`offerToQueue`). That
> wakes the pread thread blocked in `StreamingReader.poll()`, which copies the 
> data out.
> So each response costs two wake-ups (`unpark` → `pthread_cond_signal` → 
> futex) and runs on a third thread, while the
> pread thread that needs the data is idle in `poll()`.
> ### Evidence
> Secure compose cluster (Kerberos, gRPC TLS, 5 datanodes), freon `ocpr`, EC 
> rs-3-2-1024k, 64 KB preads, 8 threads,
> HDDS-17752 reuse on. async-profiler `cpu`, client side, µs of CPU per pread 
> (client total ≈ 106 µs):
> | cost | µs/pread |
> |---|---|
> | `pthread_cond_signal` (wake-ups): `SynchronousQueue.xferLifo` 5.9 + 
> `AQS.signalNext` (reader queue) 5.3 | 11.2 |
> | futex waits behind those hand-offs | 3.8 |
> | executor threads: deframe + protobuf parse | 6.0 |
> | executor threads: `offerToQueue` (enqueue + wake the reader) | 5.7 |
> | executor threads: idle / hand-off (`park`, `SynchronousQueue` poll) | 3.2 |
> The wake-ups alone are about 10% of client CPU per read. On a loaded 
> RegionServer each hand-off also adds scheduling
> latency to every read.
> ## Proposed fix
> For **positioned** reads only, make the pread thread the call's executor, as 
> HDDS-17561 does for blocking unary calls:
> - The pread `StreamingReader`, pooled since HDDS-17752, implements 
> `Executor`. While a pread is waiting, callbacks are
> queued to it and run by the pread thread inside `poll()`. That covers 
> deframing, protobuf parsing, checksum
> verification and the data copy.
> - When no pread is waiting (the reader is idle in the pool), callbacks run 
> directly on the calling thread. These are
> only `onClose`/`onReady`, with no data.
> - The response queue then disappears: `onNext` hands the response to the 
> waiting `poll()` on the same thread.
> - The **sequential** reader keeps today's executor and queue. Its pre-read 
> window relies on responses being consumed
> while the application processes earlier data. Running callbacks only when the 
> reader polls would hold back gRPC
> inbound flow control and shrink the effective pre-read.
> Same rules as HDDS-17561's `CallerRunsReplyFuture`:
> - a waiter that times out, or is interrupted, cancels the stream;
> - callbacks queued after the waiter left run directly;
> - `close()` must not leave callbacks queued on a reader nobody will poll.
> ## Compatibility / side effects
> - No protocol or datanode change. The checksum verification and protobuf 
> parsing move from a pool thread to the pread
> thread, so the pread thread does that CPU work itself instead of waiting.
> - Concurrent preads on one block already use separate readers (HDDS-17752), 
> so there is no cross-request
> head-of-line blocking.
> ## Tests
> - **Unit** (`TestStreamBlockInputStream`):
> - with a pooled reader, a response is processed on the pread thread;
> - no callback is left queued after a timeout, an interrupt or `close()`;
> - the sequential path is unchanged.
> - **Integration:** EC and RATIS positioned reads with failover, with stream 
> reuse on.
> - **Benchmark:**
> - the HDDS-17752 harnesses: `TestStreamPositionedReadBenchmark`, plus freon 
> `ocpr` on the secure compose cluster;
> - async-profiler: `SynchronousQueue.xferLifo` and 
> `StreamingReader.offerToQueue` should drop out of the pread path.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to