[ 
https://issues.apache.org/jira/browse/HDDS-17752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17752:
----------------------------------
    Labels: ozone-hbase-perf pull-request-available  (was: 
pull-request-available)

> reuse streaming readers for positioned reads instead of one stream per 
> readPositioned
> -------------------------------------------------------------------------------------
>
>                 Key: HDDS-17752
>                 URL: https://issues.apache.org/jira/browse/HDDS-17752
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: EC Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>              Labels: ozone-hbase-perf, pull-request-available
>
> h3. Problem
> With streaming read on (\{{ozone.client.stream.readblock.enable=true}}), 
> \{{StreamBlockInputStream.readPositioned}} serves every positioned read with 
> its own one-shot streaming call. The call never touches the sequential reader 
> or its pre-read buffer:
> {code:java}
> final XceiverClientGrpc client = acquireGrpcClientForReadData(factory, 
> pipeline);
> final StreamingReader reader = new StreamingReader(client);
> client.initStreamRead(blockID, reader, failedStreamingDatanodes); // new gRPC 
> call + semaphore permit
> try {
> return preadOnce(client, reader, pipeline, blockOffset, length, dst); // one 
> ReadBlock request, poll until length
> } finally {
> closeStream(reader); // half-close, release permit
> }
> {code}
> So each pread pays a full call lifecycle, which turns a streaming protocol 
> into a unary one with extra steps:
> * a client acquire, including the per-call user lookup in 
> \{{XceiverClientManager.getPipelineCacheKey}} (HDDS-17716);
> * \{{initStreamRead}}: semaphore permit, a new HTTP/2 stream, HEADERS, the 
> interceptor chain;
> * \{{StreamingReadResponse.awaitReady}}, a wait until the new stream is 
> writable;
> * on the datanode, a new \{{ServerCall}} and stream observer, plus a 
> block-file open (\{{getChunkFile}} stat and open);
> * \{{closeStream}}: half-close, trailers, \{{completeStreamRead}}, and on the 
> datanode the block file is closed again.
> HBase issues most of its reads as preads (\{{BlockIOUtils.preadWithExtra}}), 
> including every read of a newly reopened store file, so this per-pread setup 
> dominates its streaming read path.
> h3. Evidence
> HBase YCSB on EC rs-3-2, streaming read on. RegionServer async-profiler, 6 x 
> 60 s:
> * *Wall:* streaming ReadBlock waits are 60.7% of busy RPC handler time, and 
> about half of that is preads (\{{preadOnce}} -> \{{StreamingReader.poll}}). 
> \{{awaitReady}} alone is about 4% of the streaming wait.
> * *CPU, EC read path* (\{{ECBlockInputStream}} read = 100%): 
> \{{readPositioned}} is 66%. Inside it, \{{initStreamRead}} is 15%, 
> \{{closeStream}} 4.4%, and the \{{streamRead}} send 16% (5.5% of it in 
> \{{awaitReady}}).
> * *Netty* threads are 47% of RegionServer CPU in this mode: stream creation, 
> HEADERS, and a wake-up per call.
> h3. What the related changes already cover
> * *HDDS-16396* (merged) made EC preads concurrent.
> * *HDDS-16870* (merged) makes healthy EC preads reuse the long-lived 
> per-index block streams instead of per-call clones. That removes the 
> per-pread GetBlock, and it means one \{{StreamBlockInputStream}} per EC index 
> now sees every pread on its block. It does not change what that stream does 
> per pread: \{{readPositioned}} still opens a one-shot gRPC call. HDDS-16870 
> reuses the Java stream object; this Jira reuses the gRPC stream underneath 
> it. With both, an EC pread on a healthy key does no GetBlock and opens no new 
> gRPC call.
> * *HDDS-17561* (Patch Available) pools single-request streams for blocking 
> \{{sendCommand}} calls (ReadChunk and GetBlock on the classic, non-streaming 
> read path) and delivers the reply on the waiting thread. It does not touch 
> \{{initStreamRead}} or \{{streamRead}}. Its per-channel pool also cannot hold 
> ReadBlock streams as they are, because a datanode binds a ReadBlock stream to 
> one block (see below). This Jira borrows its lifecycle rules (drop on error 
> or abandon, idle trim, client-side deadline) but needs its own per-block pool.
> * *HDDS-17399* (lock-free positioned reads) sets three constraints that this 
> design follows: the pool takes no stream monitor, \{{closed}} is volatile and 
> re-checked after a reader is pooled, and an idle reader holds no client 
> permit.
> h3. Proposed fix
> Keep a small pool of idle pread readers in each \{{StreamBlockInputStream}}. 
> After a pread that read everything it asked for, its stream stays open; the 
> next pread on the block takes the idle reader and sends one more 
> \{{ReadBlock}} on it.
> * *One reader per pread at a time.* Concurrent preads use different readers. 
> This matches the datanode, which serves a stream sequentially.
> * *Reuse only after a complete, clean read.* A reader goes back to the pool 
> only if it got the full length and its stream is still open with nothing 
> unread. The datanode's \{{BlockReadCursor}} sends contiguous ranges ending 
> with the one that covers the requested end, so a complete pread leaves 
> nothing in flight. Errors, short reads, EOF and \{{OUT_OF_RANGE}} close the 
> reader. \{{preadOnce}} still checks each response's offset as a guard.
> * *Idle readers hold no client permit.* The permit is returned when a reader 
> goes idle and taken again on lease (\{{XceiverClientGrpc.resumeStreamRead}}). 
> Otherwise idle readers would eat into the 32 permits per client.
> * *Lock-free pool:* a \{{ConcurrentLinkedDeque}} used most-recently-used 
> first, with a CAS-bounded count.
> * *Close, unbuffer and refresh close all idle readers.* A pool generation 
> advances on \{{unbuffer()}} and block-info refresh. A pread that started in 
> an older generation, or that finishes after \{{close()}}, closes its reader 
> instead of pooling it, so no reader of an old pipeline survives a refresh.
> * *A failed reused stream does not blame the datanode.* An idle stream may 
> have broken while pooled. Its first failure is retried once on a new stream 
> without adding the datanode to \{{failedStreamingDatanodes}}, and the other 
> idle readers are dropped. Timeouts and interrupts are not retried, so a hung 
> datanode still fails over after one read timeout.
> * *Bounded and time-limited*, behind a flag:
> ||Config||Default||Meaning||
> |ozone.client.stream.read.pread.reuse.enabled|false|turn the reuse on|
> |ozone.client.stream.read.pread.max-idle-readers|2|idle readers kept per 
> block stream|
> |ozone.client.stream.read.pread.idle-timeout|1s|close idle readers after 
> this, well below the datanode's 1 min stream file idle timeout|
> h3. Constraint: one block per stream
> The datanode opens the per-stream block file on the first \{{ReadBlock}} and 
> reuses it without checking the block ID of later requests 
> (\{{KeyValueHandler.readBlockImpl}} opens \{{blockFile}} only if it is not 
> open yet). So a reader cannot be shared across blocks or across 
> \{{StreamBlockInputStream}} instances, and the pool lives in the block 
> stream, not in \{{XceiverClientGrpc}}.
> h3. Rejected alternatives
> * *Serve preads from the sequential reader.* Its queue holds pre-read data 
> for the cursor position, so interleaved pread responses would mean draining 
> or discarding that window.
> * *Multiplex concurrent preads on one stream.* The datanode serves a stream 
> sequentially, and responses carry offsets but no request id.
> * *Unary ReadChunk for small preads.* With HDDS-17561 a ReadChunk no longer 
> opens a gRPC stream, but it needs the GetBlock chunk list, which the 
> streaming path otherwise avoids, and it mixes two read protocols on one block 
> stream.
> h3. Results
> A/B with only the flag toggled, same build. Secure compose cluster (Kerberos, 
> gRPC TLS, 5 datanodes), \{{ozone freon ocpr}} on 48 MB keys, 3 repetitions 
> per arm, 108 runs, 0 failures:
> ||Key||Read size||Threads||preads/s off -> on||Speedup||p99 ms off -> 
> on||Client CPU per read, us||
> |EC rs-3-2|4 KB|8|14206 -> 24089|1.69x|2.07 -> 1.24|233 -> 118|
> |EC rs-3-2|64 KB|1|2454 -> 3472|1.42x|0.82 -> 0.47|474 -> 244|
> |EC rs-3-2|64 KB|8|11614 -> 18052|1.54x|2.46 -> 1.51|291 -> 163|
> |EC rs-3-2|64 KB|32|20688 -> 27628|1.34x|5.19 -> 4.01|212 -> 139|
> |EC rs-3-2|1 MB|8|3057 -> 3382|1.11x|5.50 -> 5.14|1196 -> 1018|
> |RATIS/THREE|4 KB|8|14435 -> 23997|1.66x|2.11 -> 1.37|235 -> 116|
> |RATIS/THREE|64 KB|8|11620 -> 18113|1.55x|1.94 -> 1.45|289 -> 162|
> |RATIS/THREE|1 MB|8|3068 -> 3401|1.11x|5.86 -> 5.22|1197 -> 1019|
> Profiles at 64 KB, 8 threads: client CPU per pread falls from 187 to 106 us, 
> \{{initStreamRead}} and \{{closeStream}} drop by about 90%, datanode 
> block-file opens drop 96%, and reader-thread \{{awaitReady}} wait falls from 
> 18% to 0.9% of wall time. HBase reads 64 KB blocks with preads, so the 64 KB 
> rows are the relevant ones.
> h3. Compatibility / side effects
> * None with the flag off (the default).
> * No protocol or datanode change. Datanodes already accept several 
> \{{ReadBlock}} requests per stream; the sequential reader relies on this.
> * With the flag on, each idle reader keeps a datanode stream and an open 
> block file on that datanode for up to the idle timeout, at most 2 per block 
> stream. A client actively reading 1,000 EC rs-3-2 block groups can hold up to 
> about 6,000 extra open block files across the datanodes for up to 1 s.
> * An idle reader keeps its \{{XceiverClientGrpc}} referenced, so 
> \{{XceiverClientManager}} cannot close that client while the reader is idle. 
> The idle timeout bounds this.
> h3. Follow-ups (out of scope)
> * *HDDS-17867:* deliver streaming pread replies on the waiting thread, 
> removing the two thread hand-offs per response (about 15-20% of client CPU 
> per pread with this reuse on).
> * *Datanode re-opens the block file when a stream's block ID changes.* That 
> would allow a per-channel pool shared across blocks, like HDDS-17561's, but 
> it needs both sides and a version check.
> * *Leftovers of the classic path, for HDDS-17561:* traced blocking calls 
> still use one stream per call, because the datanode interceptor takes the 
> span from call headers. Pooling is also skipped whenever any OpenTelemetry 
> span is current. These are review items there, not part of this Jira.
> * *HDDS-17716:* the per-call user lookup in \{{getPipelineCacheKey}} is still 
> paid on every lease.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to